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24th  ANNUAL  NATIONAL  TEST  &  EVALUATION  CONFERENCE 
“Test  and  Evaluation  of  Autonomous  Systems  &  The  Role  of  the  T&E  Community  in  the  Requirements  Process” 

PALM  SPRINGS,  CA 
FEBRUARY  25  -  28,  2008 


Agenda 


MONDAY.  FEBRUARY  25.  2008 


SESSION  A 

“Combinatorial  Testing:  Required  Knowledge  in  a  World  Where  Traversing  Every  Possible  Path  is  Impossible,”  Dr.  Mark 
Kiemele,  President  and  Cofounder,  Air  Academy  Associates 

SESSION  B 

“System  Evaluation  of  UAVs,”  Dr.  Juan  A.  Vitali,  Chief,  T&E,  Joint  PEO  Chem  Bio  Defense 


TUESDAY.  FEBRUARY  26.  2008 


SESSION  A 

Unmanned  Systems:  What’s  New,  What’s  Not? 

•  Mr.  Gene  Fraser,  Vice  President,  Northrop  Grumman  Unmanned  Systems 

•  Mr.  Allen  Nease,  Chief,  Robotics,  Tyndall  AFB 

Government  Keynote  Address:  Hon  Charles  E.  McQueary,  Director  OT&E,  OSD 
SESSION  B 

•  Maj  Gen  Stephen  T.  Sargeant,  USAF,  Commander,  Air  Force  Operational  T&E  Command  (AFOTEC) 

•  RDML  Stephen  Voetsch,  USN,  Commander,  Navy  Operational  T&E  Force  (OPTEVFOR) 

•  Dr.  James  Streilein,  Technical  Director,  Army  T&E  Command  (ATEC) 

•  Mr.  Stephen  Daly,  Deputy  Director,  ODOT&E,  OSD 

Luncheon  Address:  “Motivating  Young  Americans  to  Pursue  Robotics  Technology,”  Mr.  Marco  Ciavolino,  President,  Enktesis,  LLC 

SESSION  C 

•  "Unique  Range  Challenges  of  UAV/RPV/UUV  Test  and  Evaluation,”  RDML  David  Dunaway,  USN,  Commander,  Naval  Air  Warfare  Center,  Weapons 
Division,  China  Lake,  AIR  5.0  Assistant  Commander  for  T&E  “Airspace  Management  Issues  for  UAVs,  RPVs,”  Mr.  Jeffrey  “Goldy”  Goldfi  nger,  ASTM 
Committee  F38  Vice  Chairman,  L-3  Communications,  Inc. 

CONCURRENT  FOCUS  SESSION  D 


T&E  ROLE  IN  SYSTEM  REQUIREMENTS 

•  “Interoperability  Testing  in  a  Net  Centric  Environment;  The  Role  of  the  Test  Community  in  Development  and  Assessment  of  System-of-System 
Requirements,”  Mr.  Frank  Alvidrez,  MTSI  Corporation 

•  “Automatic  Generation  of  Requirement  Specif!  cations  (Verifi  cation),”  Mrs.  Monika  Morgan,  General  Dynamics  Land  Systems 

•  “Improving  Requirements  &  Testing  Process  for  Rapidly  Fielded  Unmanned  Aircraft  System  Program,”  Mr.  Jeremy  S.  Dusina,  U.S.  Army  Evaluation 
Center 

•  “Individual  Protection  in  a  CBRN  Environment:  Case  Studies  of  the  Role  of  T&E  in  Requirements  Generation,”  Ms.  Karen  McGrady,  JPM  Individual 
Protection 
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T&E  POLICY  ISSUES 

•  “Application  of  MIL-HDBK-189  Reliability  Growth  Analysis  (RGA)  on  Mobile  Gun  System  (MGS)  Product  Verification  Test  (PVT),”  Dr.  Dmitry 
Tananko,  General  Dynamics  Land  Systems 

•  “Innovative  Acquisition  Methodologies  in  Support  of  Test  &  Evaluation  Objectives,”  Mr.  Frank  Parker,  General  Services  Administration 

•  “Applicability  of  Evolutionary  Strategies  for  Acquisition  of  Autonomous  Systems,”  Dr.  Ebru  Sarigol,  STM  AS,  Turkey 

•  “Unmanned  Systems  Safety  Guide  for  DoD  Acquisition,”  Mr.  Michael  Demmick,  NOSSA,  N3 14,  WSESRB 

•  “Unmanned  Vehicle  Synthetic  Environment,”  Mr.  Angelo  Prevete,  Lockheed  Martin  Corporation 

OT&E  CHALLENGES 

•  “Operational  Testing  (OT)  and  Unmanned  Aerial  Vehicles,”  Mr.  Curt  Cook,  ODOT&E,  OSD 

•  “Test  Pilot’s  Role  in  Flight  Test  of  Unmanned  Vehicles,”  Mr.  Roy  Martin,  Northrop  Grumman  Corporation 

•  “Testing  in  a  Joint  Environment  Measures  Framework,”  Mr.  Max  Lorenzo,  JTEC 

•  “Challenges  for  Testing  in  Defense  System  of  Systems  Environment,”  Dr.  Judith  Dahmann,  MITRE  Corporation 

•  “Evaluating  Autonomy  of  UAVs,”  Dr.  Herbert  Hecht,  SoHar,  Inc. 


WEDNESDAY.  FEBRUARY  27.  2008 


Former  DOT&E’s  Weigh-in  on  T&E  Issues 

•  Hon  Philip  E.  Coyle,  III,  Director  OT&E  (1994  -  2001) 

•  Hon  Thomas  Christie,  Director  OT&E  (2001  -  2006) 

NDIA  Industrial  Committee  on  Operational  T&E  (ICOTE)  Update  from  the  Chairman,  Mr.  Larry  Graviss,  President,  Eagle  Engineering,  Inc. 

SESSION  F  -  CONFERENCE  TOWN  MEETING 

Chairs:  Dr.  Paul  H.  Deitz,  HRED,  ARL  and  Mr.  Britt  Bray,  DRC  Corporation 

SESSION  G  -  GOVERNMENT  STUDIES  /  REPORTS  IMPACTING  THE  T&E  COMMUNITY 

•  “Opportunities  to  Optimize  the  Operational  Use  of  Unmanned  Systems  and  Gain  Synergy  in  Developing  New  Capabilities,” 

Ms.  Sharon  Pickup,  GAO,  DC  and  Mr.  Michael  Sullivan,  GAO,  Dayton 

CONCURRENT  FOCUS  SESSION  H 

T&E  ANALYTICAL  APPROACHES 

•  Live,  Virtual,  and  Constructive  Simulation  Use  for  Unmanned  Vehicle  Requirements  and  T&E,”  Mr.  Gary  Kollmorgen, 

Referenda 

•  “The  Four  Element  Framework:  Progress  Towards  an  Integrated  and  Mission-based  T&E  Strategy,”  Mr.  Chris  Wilcox,  Army  Evaluation  Center 

•  “An  Enterprise  Environment  for  Information  Assurance  /  Computer  Network  Defense  T&E,”  Mr.  Steve  Moore,  Booz 
Allen  Hamilton 

•  “Applying  Design  of  Experiments  Methodology  to  Sortie  Generation  Rate  T&E,”  Mr.  Joseph  Tribble,  AVW  Technologies 
TEST  INSTRUMENTATION  /  DATA  COLLECTION  /  ARCHITECTURE 

•  “TRIZ  and  Measurement  and  Detection  Problems,”  Mr.  Michael  Slocum,  Air  Academy  Associates 

•  "Time  Space  Position  Information  (TSPI)  T&E  Instrumentation  Technology  Investment  Plan  -  Ensuring  T&E  Ranges  Can 

Meet  the  Test  Requirements  of  the  Future,”  Mr.  Dick  Dickson,  Tybrin  Corporation44 Semi- Autonomous  Delivery  Transport  Vehicle  for  Undersea  Sensors 
System  Integration  Testing  Results  Lessons-Leamed,”  Mr.  Pete  Reinagel,  SYS  Technologies 

•  “Test  and  Training  Enabling  Architecture,  TENA,  an  Important  Component  in  Joint  Mission  Environment  Test  Capability 
(JMETC)  Successes,”  Mr.  Gene  Hudgins,  BAE  Systems 

•  “Preserving  Critical  Knowledge,”  Mr.  Roy  Weber,  VSE  Corporation 

T&E  ROLE  IN  SYSTEM  REQUIREMENTS 

•  “Warfare  Systems  Engineering  Challenges  and  T&E  Approaches,”  Mr.  Robert  Connemey,  Naval  Undersea  Warfare  Center  Division,  Newport 

•  “Integrating  T&E  into  DoD  Acquisition  Contracts,”  Ms.  Darlene  Mosser-Kemer,  OUSD  (AT&L) 

•  “Improving  T&E  Participation  in  the  Requirements  Generation  Process,”  Mr.  Keith  Montgomery,  Lockheed  Martin 
Corporation 


THURSDAY.  FEBRUARY  28.  2008 


“UAVs  On-campus  R&D,”  Dr.  Steve  Hottman,  Physical  Sciences  Lab,  Associate  Dean,  New  Mexico  State  University 
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SESSION  I  -  REPORTS  FROM  THE  COMBAT  ZONE:  OIF  &  OEF 

•  “ATEC  Forward  Operational  Assessment  Team,”  COL  Mark  Mills,  USA,  U.S.  Army  ATC 

•  “Lessons  Learned  in  Fielding  a  UAS  RQ4A  Global  Hawk  in  the  Combat  Theater,”  Mr.  Sam  McKeehan,  Northrop  Grumman  Corporation 
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PRESENTERS  INCLUDE: 


Hon  Philip  Coyle, 
Former  DOT&E 


Mr.  Dick  Rutan, 
Aviator  and 
Entrepreneur 

Sponsored  by  the  T&E 
Division,  NDIA  and 
Co-sponsored  by 
ASTM  and  the  Aircraft 
Survivability,  Combat 
Survivability  and 
Robotics  Divisions,  NDIA 
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3  Gen  Larry  Welch, 
USAF  (Ret) 

Hon  Thomas 
Christie,  Former 
DOT&E 


CONFERENCE  AGENDA 


MONDAY,  FEBRUARY  25, 2008 


9:00AM  -  5:00PM  Conference  Registration  -  Horizon  Foyer 

TUTORIAL  SESSIONS 


►  Plaza  AB 


►  Plaza  CD 


SESSION  A 

SESSION  B 

1:30PM 

1)  “Combinatorial  Testing:  Required 

Knowledge  in  a  World  Where  Traversing 

Every  Possible  Path  is  Impossible,”  Dr.  Mark 
Kiemele,  President  and  Cofounder,  Air 

Academy  Associates 

2)  “Survivability  of  Unmanned  Systems,”  Dr.  Albert 
Moussa,  President,  BlazeTech,  Inc. 

3:00PM 

Afternoon  Break 

Afternoon  Break 

3:15PM 

1  Cont.)  “Combinatorial  Testing:  Required 
Knowledge  in  a  World  Where  Traversing 

Every  Possible  Path  is  Impossible,”  Dr.  Mark 
Kiemele,  President  and  Cofounder,  Air 
Academy  Associates 

3)  “System  Evaluation  of  UAVs,”  Dr.  Juan  A.  Vitali, 
Chief,  T&E,  Joint  PEO  Chem  Bio  Defense 

Tutorial  Sessions  are  provided  free  of  charge  to  all  those  registering  for  the  conference.  Your  attendance  at 
these  valuable  tutorials  is  encouraged. 

5:00PM  -  6:00PM  Evening  Reception  /  Display  Area  Open  -  Horizon  Ballroom 

6:00PM  Display  Area  Closed  for  the  Day  /  Conference  Adjourned  for  the  Day 


TUESDAY,  FEBRUARY  26, 2008 

7:00AM  -  5:00PM  Conference  Registration  Open  -  Horizon  Foyer 

7:00AM  -  8:00AM  Continental  Breakfast  in  Display  Area  -  Horizon  Ballroom 

SESSION  A 

Chair:  Mr.  James  O’Bryon,  Chairman,  NDIA  T&E  Division 

8:00AM  Call  to  Order  and  Remarks  -  Plaza  Ballroom,  Mr.  Sam  Campagna,  Director, 

Operations,  NDIA 

8:05AM  Tribute  to  our  Nation  and  Fighting  Forces,  National  Anthem 

8:10AM  Welcome  and  Conference  Introductory  Remarks,  Mr.  James  O’Bryon,  Chairman, 

NDIA  T&E  Division 


FEBRUARY  25-28,  2008 


CONFERENCE  KICKOFF  CAUCUS 


8:15AM 

Unmanned  Systems:  What’s  New,  What’s  Not? 

Chair:  VADM  Joseph  Dyer,  USN  (Ret),  Executive  Vice  President  &  General 
Manager,  iRobot  Corporation 

►  Mr.  Gene  Fraser,  Vice  President,  Northrop  Grumman  Unmanned  Systems 

►  Mr.  Michael  Gaulke,  Chairman  and  CEO,  Exponent  Corporation 

►  RADM  Timothy  Heely,  USN,  PEO  Unmanned  Systems,  NAVAIR 

►  Mr.  Allen  Nease,  Chief,  Robotics,  Tvndall  AFB 

9:15  AM 

Government  Keynote  Address:  Hon  Charles  E.  McQueary,  Director  OT&E,  OSD 

9:45AM 

Military  Perspectives  Keynote  Address:  General  Larry  D.  Welch,  USAF  (Ret), 
President,  IDA,  Former  Chief  of  Staff,  USAF 

10:15AM 

Morning  Break  &  Networking  in  Display  Area  -  Horizon  Ballroom 

SESSION  B 

Chair:  Dr.  Juan  A.  Vitali,  Chief,  T&E,  Joint  PEO  Chem  Bio  Defense 


10:45  AM 

OTAs  Roundtable:  “Applying  the  T&E  Requirements  Process  to 

Unmanned  /  Autonomous  Vehicles” 

►  Maj  Gen  Stephen  T.  Sargeant,  USAF,  Commander,  Air  Force  Operational  T&E 
Command  (AFOTEC) 

►  RDML  Stephen  Voetsch,  USN,  Commander,  Navy  Operational  T&E  Force 
(OPTEVFOR) 

►  Col  Michael  S.  Bohn,  USMC,  Commander,  U.S.  Marine  Corps  Operational  T&E 
Activity  (MCOTEA) 

►  Dr.  James  Streilein,  Technical  Director,  Army  T&E  Command  (ATEC) 

►  Mr.  Stephen  Daly,  Deputy  Director,  ODOT&E,  OSD 

12:15PM -1:15PM 

Luncheon  -  Horizon  Ballroom:  “Motivating  Young  Americans  to  Pursue  Robotics 
Technology,”  Mr.  Marco  Ciavolino,  President,  Enktesis,  LLC 

A  Roomba  will  be  given  away  to  benefit  students  of  TechBrick  Robotics.  TeckBrick 
is  a  robotics  club  formed  in  2003  for  home  schoolers  in  the  Baltimore,  MD  area. 
This  year,  TechBrick  has  four  U.S.  FIRST  competition  teams  with  a  total  of  30 
students  and  5  adult  coaches.  Over  the  past  4  years,  TechBrick  has  helped  train 
more  than  40  young  engineers-to-be  through  the  robotics  program  offered  by 
FIRST.  To  view  more  information,  please  visit  www.techbrick.com. 

SESSION  C 

Chair:  Dr.  Paul  H.  Deitz,  Acting  Director,  HRED,  ARL 


1:30PM 

“Unique  Range  Challenges  of  UAV/RPV/UUVTest  and  Evaluation,”  RDML  David 
Dunaway,  USN,  Commander,  Naval  Air  Warfare  Center,  Weapons  Division,  China 
Lake,  AIR  5.0  Assistant  Commander  for  T&E 

1:55PM 

“Airspace  Management  Issues  for  UAVs,  RPVs,”  Mr.  Jeffrey  “Goldy”  Goldfinger, 
ASTM  Committee  F38  Vice  Chairman,  L-3  Communications,  Inc. 

2:20PM 

Afternoon  Break  &  Networking  in  Display  Area 

-  Horizon  Ballroom 

PALM  SPRINGS,  CA 


CONCURRENT  FOCUS  SESSION  D 


►  Plaza  AB 


►  Plaza  CD 


►  Palm  Canyon  AB 


D1)T&E  ROLE  IN  SYSTEM 
REQUIREMENTS 

Chair:  Dr.  Lowell  Tonnessen, 
IDA 

D2) T&E  POLICY  ISSUES 

Chair:  Mr.  Dick  Dickson,  Tybrin 
Corporation 

D3)  OT&E  CHALLENGES 

Chair:  RADM  “Bert”  Johnston, 
USN  (Ret),  Wyle  Laboratories 

2:30PM 

“How  Early  Involvement  Can 
Benefit  the  Requirements 
Generation  Process  for  UAV 
Ground  Checkout  and 
Maintenance,”  Mr.  Steven 
Madison,  Northrop  Grumman 
Corporation 

“Application  of  MIL-HDBK-189 
Reliability  Growth  Analysis 
(RGA)  on  Mobile  Gun  System 
(MGS)  Product  Verification  Test 
(PVT),”  Dr.  Dmitry  Tananko, 
General  Dynamics  Land 

Systems 

“Operational  Testing  (OT)  and 
Unmanned  Aerial  Vehicles,” 

Mr.  Curt  Cook,  ODOT&E,  OSD 

3:00PM 

“Interoperability  Testing  in  a 

Net  Centric  Environment;  The 
Role  of  the  Test  Community  in 
Development  and  Assessment 
of  System-of-System 
Requirements,”  Mr.  Frank 
Alvidrez,  MTSI  Corporation 

“Innovative  Acquisition 
Methodologies  in  Support  of  Test 
&  Evaluation  Objectives,” 

Mr.  Frank  Parker,  General 
Services  Administration 

“Test  Pilot’s  Role  in  Flight  Test 
of  Unmanned  Vehicles,” 

Mr.  Roy  Martin,  Northrop 
Grumman  Corporation 

3:30PM 

“Automatic  Generation  of 
Requirement  Specifications 
(Verification),”  Mrs.  Monika 
Morgan,  General  Dynamics 
Land  Systems 

“Applicability  of  Evolutionary 
Strategies  for  Acquisition  of 
Autonomous  Svstems,”  Dr.  Ebru 
Sariaol.  STM  AS.  Turkey 

“Testing  in  a  Joint  Environment 
Measures  Framework,”  Mr.  Max 
Lorenzo,  JTEC 

4:00PM 

“Improving  Requirements  & 
Testing  Process  for  Rapidly 
Fielded  Unmanned  Aircraft 
System  Program,”  Mr.  Jeremy 
S.  Dusina,  U.S.  Army 
Evaluation  Center 

“Unmanned  Systems  Safety 

Guide  for  DoD  Acquisition,” 

Mr.  Michael  Demmick,  NOSSA, 
N314,  WSESRB 

“Challenges  for  Testing  in 
Defense  System  of  Systems 
Environment,”  Dr.  Judith 
Dahmann,  MITRE  Corporation 

4:30PM 

“Individual  Protection  in  a 
CBRN  Environment:  Case 
Studies  of  the  Role  of  T&E  in 
Requirements  Generation,” 

Ms.  Karen  McGradv.  JPM 
Individual  Protection 

“Proteus  Testbed  Utilized  to 

Flight  Test  New  Global  Hawk 
Radar,”  Mr.  David  Brown, 

Northrop  Grumman  Corporation 

“Evaluating  Autonomy  of  UAVs,” 
Dr.  Herbert  Hecht,  SoHar,  Inc. 

5:00PM 

“Unmanned  Vehicle  Synthetic 
Environment,”  Mr.  Angelo 

Prevete,  Lockheed  Martin 
Corporation 

“Deployability  and  Survivability 
Testing  for  the  Deployable 
Autonomous  Distributed 

System  (DADS),”  Mr.  Steve 
Whiteside,  SPAWAR 

FEBRUARY  25-28,  2008 


5:30PM  Display  Area  Closed  for  the  Day  /  Conference  Adjourned  for  the  Day 

6:30PM  -  8:30PM  Walter  W.  Hollis  Honors  Banquet  -  Horizon  Ballroom 

Presentation  of  the  Walter  W.  Hollis  Award  for  outstanding  lifetime  achievement  in 
defense  Test  &  Evaluation. 


Recipient: 

Dr.  Paul  H.  Deitz,  Acting  Director  HRED,  ARL,  APG,  MD 

In  his  early  RDT&E  work  at  the  Ballistic  Research  Laboratories,  Dr.  Deitz 
distinguished  himself  in  laser  scintillation  testing  and  eye-damage  modeling 
studies  and  later  in  evaluations  of  smart  weapons.  Over  the  ensuing  two 
decades,  his  seminal  work  in  ballistic  live-fire  simulation  and  testing  has 
contributed  to  improved  vulnerability/survivability  of  our  front-line  combat  platforms. 

Dr.  Deitz  is  a  prolific  writer,  publishing  numerous  papers  examining  the  detailed 
measures  of  simulation  and  testing,  providing  clarity  and  mutual  synergism  to  the 
benefit  of  both  disciplines.  Over  the  past  decade,  Dr.  Deitz  has  served  as  the 
Technical  Director  of  AMS AA  and  the  Acting  Director  of  the  Survivability/  Lethality 
Analysis  Directorate,  ARL. 


He  earned  a  BA  in  Physics  from  Gettysburg  College  and  an  MS  and  Ph.D.  in 
Electrical  Engineering  from  the  University  of  Washington. 


Guest  Speaker: 

Mr.  Dick  Rutan,  Aviator  and  Entrepreneur 

Mr.  Dick  Rutan  made  history  in  December  of  1986  after  completing  Voyager’s 
nine  day,  three  minute  and  forty-four  second  flight  around  the  world,  non-stop  and 
non-refueled,  setting  world  records  that  remain  unchallenged  today.  In  July  of 2002, 
Mr.  Rutan  was  inducted  into  the  National  Aviation  Hall  of  Fame  in  Dayton,  OH. 

As  a  youth,  he  was  fascinated  by  airplanes  and  flying  and  made  his  first  flight  at 
age  six.  He  received  his  solo  pilot’s  license  and  driver’s  license  on  his  16th 
birthday;  the  first  possible  moment  he  could  be  pilot  in  command. 

Retired  from  the  United  States  Air  Force  as  a  Lieutenant  Colonel,  Mr.  Rutan  flew 
325  missions  in  Vietnam,  105  of  them  as  a  member  of  the  Super  Sabre  FAC,  a 
high-risk  operation  commonly  known  as  the  “MISTY’s.”  He  was  hit  by  enemy 
ground  fire  on  his  last  mission  and  was  forced  to  eject  from  his  burning  F-100,  to 
be  rescued  later. 


Before  retiring  from  the  Air  Force  in  1978,  Mr.  Rutan  was  awarded  the  Silver 
Star,  five  Distinguished  Flying  Crosses,  16  Air  Medals  and  a  Purple  Heart.  Today, 
he  travels  the  world  lecturing  about  his  flights  and  aviation’s  future,  stressing,  “If 
you  can  dream  it,  you  can  do  it!” 


WEDNESDAY,  FEBRUARY  27, 2008 


7:00AM -5:00PM 
7:00AM  -  8:00AM 


Conference  Registration  Open  -  Horizon  Foyer 

Continental  Breakfast  in  the  Display  Area 

-  Horizon  Ballroom 


PALM  SPRINGS,  CA 


SESSION  E 

Chair:  Dr.  Mark  Kiemele,  President  and  Cofounder,  Air  Academy  Associates 

8:00AM  Call  to  Order  and  Remarks  -  Plaza  Ballroom,  Mr.  Sam  Campagna, 

Director,  Operations,  NDIA 

8:05AM  “Ground  Robotics,  Where  Are  We  Headed?”  Ms.  Ellen  Purdy,  Director, 

Joint  Ground  Robotics,  OUSD  (AT&L) 

8:35AM  Former  DOT&E’s  Weigh-in  on  T&E  Issues 

►  Hon  Philip  E.  Coyle,  III,  Director  OT&E  (1994  -  2001) 

►  Hon  Thomas  Christie,  Director  OT&E  (2001  -  2006) 

9:35AM  NDIA  Industrial  Committee  on  Operational  T&E  (ICOTE)  Update  from  the 

Chairman,  Mr.  Larry_  Graviss,  President,  Eagle  Engineering,  Inc. 

10:00AM  Morning  Break  &  Networking  in  the  Display  Area  -  Horizon  Ballroom 

SESSION  F  -  CONFERENCE  TOWN  MEETING 

Chairs:  Dr.  Pau[  H.  Deitz ,  HRED,  ARL  and  Mr.  Brjtt  Bray,  DRC  Corporation 

10:30AM  The  Role  of  T&E  in  the  Requirements  Process 

►  Brig  Gen  Greg  Feest,  USAF,  J8,  Joint  Chiefs  of  Staff 

►  Mr.  Allan  M.  Resnick,  Director  of  Requirements  Integration  Directorate, 
Army  Capabilities  Integration  Center  (ARCIC),  Ft.  Monroe 

►  Mr.  James  Ruma,  Vice  President,  Engineering  Programs,  GDLS 

►  RADM  Bill  McCarthy,  USN  (Ret),  DDOT&E  /  Net  Centric  Warfare 

►  Col  Michael  S.  Bohn,  USMC,  Commander,  U.S.  Marine  Corps 
Operational  T&E  Activity  (MCOTEA) 

The  Town  Meeting  is  a  unique  forum  used  by  the  T&E  Division  to  enable 
free  and  candid  discussion  of  important,  sometimes  sensitive  and 
controversial  subjects.  All  comments  made  during  the  town  meeting  by 
both  panelists  and  the  audience  are  off  the  record  and  not  for  attribution 
to  encourage  candor  and  give-and-take  on  the  topic  at  hand. 

The  issue  being  addressed  this  year  is  the  role  that  the  T&E  community 
has,  does  and  should  play  in  the  development,  establishment  and 
assessment  of  system  requirements.  Policies  over  the  years  have  tended 
to  keep  the  T&E  community  somewhat  isolated  from  the  requirements 
process  due  to  the  perceived  potential  conflict  of  interest. 

With  the  growing  emphasis  on  more  rapid  fielding,  spiral  development, 
acquisition  reform,  escalating  costs  and  a  host  of  other  pressures  on  the 
acquisition  process,  a  renewed  discussion  on  the  role  of  T&E  in  this 
process  is  overdue.  Our  distinguished  participants  will  share  their  candid 
thoughts  and  debate  this  issue  with  each  other,  as  well  as  fielding 
questions  from  the  audience. 

Members  of  the  media  are  excluded  from  this  session. 
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12:00PM -1:00PM 


Annual  Tester  of  the  Year  Awards  Luncheon 

-  Horizon  Ballroom 


r 


2007  Tester  of  the  Year  Award  Recipients: 

OSD 

Military:  COL  Kurt  Lambert,  USA 
Civilian:  Ms.  Darlene  Mosser-Kerner 
Contractor:  Mr.  Brent  Crabtree 

OSD  Special  Category 

Mine  Resistant  Ambush  Protected  Vehicle 

Joint  Test  Team 

U.S.  Air  Force 

Military:  Lt  Col  Shaun  McGrath,  USAF 
Civilian:  Mr.  Clyde  Shook 
Contractor:  Mr.  Derrell  Parker 

U.S.  Army 

Military:  MAJ  Robert  McClintock,  USA 
Civilian:  Mr.  Gregory  Brewer 
Contractor:  Ms.  Lori  Keaty 

U.S.  Marine  Corps 

Military:  Maj  Stephen  Augustin,  USMC 
Civilian:  Mr.  Paul  Johnson 

U.S.  Navy 

Military:  LT  Thomas  Kneale,  USN 
Civilian:  Mr.  Mark  Koromhas 
Contractor:  Ms.  Susan  Smola 

SESSION  G  -  GOVERNMENT  STUDIES /REPORTS  IMPACTING 
THE  T&E  COMMUNITY 

Chair:  Mr.  William  Yeakel,  President,  ORSA  Corporation 

1:15PM  Defense  Science  Board  (DSB)  Study  Report 

“Terms  of  Reference,”  Mr.  Christopher 
DiPetto,  Executive  Secretary,  DSB,  OUSD 
(A&T) 

“Synopsis  of  Results  and  Potential  Impact  of 
the  DSB  Report,”  Mr.  Charles  “Pete”  Adolph, 
Chair,  DSB  Study 

2:00PM  Report  of  GAO  Study  of  UAVs 

“Opportunities  to  Optimize  the  Operational 
Use  of  Unmanned  Systems  and  Gain 
Synergy  in  Developing  New  Capabilities,” 

Ms.  Sharon  Pickup  GAO,  DC  and  Mr.  Michael 
Sullivan,  GAO,  Dayton 
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2:30PM 


Afternoon  Break  &  Networking  in  Display  Area  -  Horizon  Ballroom 


CONCURRENT  FOCUS  SESSION  H 

►  Plaza  AB  ►  Plaza  CD  ►  Palm  Canyon  AB 


H1)T&E  ANALYTICAL 
APPROACHES 

Chair:  Dr.  Anne  Hillegas,  Applied 
Research  Associates,  Inc. 

H2)TEST  INSTRUMENTATION  / 
DATA  COLLECTION  / 
ARCHITECTURE 

Chair:  Mr.  Britt  Bray,  DRC  Corporation 

H3)  T&E  ROLE  IN  SYSTEM 
REQUIREMENTS 

Chair:  Mr.  Charles  Larson, 
SURVICE  Engineering 

3:00PM 

“Live,  Virtual,  and  Constructive 
Simulation  Use  for  Unmanned 
Vehicle  Requirements  and 
T&E,”  Mr.  Gary  Koll morgen, 
Referenda 

“TRIZ  and  Measurement  and 
Detection  Problems,”  Mr.  Michael 
Slocum,  Air  Academy  Associates 

“Suitability-Effectiveness 
Trade-Off  Economics,” 

Dr.  Donald  Gaver,  Naval 
Postgraduate  School, 

Operations  Research 
Department 

3:25PM 

“Root  Cause  Analysis  & 
Evaluation  of  the  Stryker 
Remote  Weapon  Station 

Inhibit  Zone  Violation  Using  a 
Six  Sigma  Approach,”  Mr.  David 
Cafagna,  General  Dynamics 
Land  Systems 

“Time  Space  Position  Information 
(TSPI)  T&E  Instrumentation 
Technology  Investment  Plan 
-  Ensuring  T&E  Ranges  Can 

Meet  the  Test  Requirements  of 
the  Future,”  Mr.  Dick  Dickson, 
Tybrin  Corporation 

“Warfare  Systems  Engineering 
Challenges  and  T&E 
Approaches,”  Mr.  Robert 
Connerney,  Naval  Undersea 
Warfare  Center  Division, 
Newport 

3:50PM 

“The  Four  Element  Frame¬ 
work:  Progress  Towards  an 
Integrated  and  Mission-based 
T&E  Strategy,”  Mr.  Chris 

Wilcox,  Army  Evaluation 

Center 

“Semi-Autonomous  Delivery 
Transport  Vehicle  for  Undersea 
Sensors  System  Integration 
Testing  Results  Lessons-Learned,” 
Mr.  Pete  Reinagel,  SYS 
Technologies 

“Integrating  T&E  into  DoD 
Acquisition  Contracts,”  Ms. 
Darlene  Mosser-Kerner,  OUSD 
(AT&L) 

4:15PM 

“An  Enterprise  Environment 
for  Information  Assurance  / 
Computer  Network  Defense 
T&E,”  Mr.  Steve  Moore,  Booz 
Allen  Hamilton 

“Test  and  Training  Enabling 
Architecture,  TENA,  an  Important 
Component  in  Joint  Mission 
Environment  Test  Capability 
(JMETC)  Successes,”  Mr.  Gene 
Hudgins,  BAE  Systems 

“Incorporating  T&E  into  Your 
Requirements  Process,”  Mr. 
David  Henry,  Lockheed  Martin 
Corporation 

4:40PM 

“Applying  Design  of 
Experiments  Methodology  to 
Sortie  Generation  Rate  T&E,” 
Mr.  Joseph  Tribble,  AVW 
Technologies 

“Preserving  Critical  Knowledge,” 
Mr.  Roy  Weber,  VSE  Corporation 

“Improving  T&E  Participation  in 
the  Requirements  Generation 
Process,”  Mr.  Keith 

Montgomery,  Lockheed  Martin 
Corporation 

5:05PM 


Display  Area  Closed  for  the  Day  /  Conference  Adjourned  for  the  Day 
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THURSDAY,  FEBRUARY  28,  2008 


7:00AM -12:00PM 

Conference  Registration  Open  -  Horizon  Foyer 

7:00AM  -  8:00AM 

Continental  Breakfast  in  Display  Area  -  Horizon  Ballroom 

8:00AM 

Call  to  Order  and  Remarks  -  Plaza  Ballroom,  Mr.  Sam  Campagna,  Director, 
Operations,  NDIA 

8:05AM 

“UAVs  On-campus  R&D.”  Dr.  Steve  Hottman.  Phvsical  Sciences  Lab. 

Associate  Dean,  New  Mexico  State  University 

SESSION  I  -  REPORTS  FROM  THE  COMBAT  ZONE:  OIF  &  OEF 


8:30AM 

“Prevention  of  Fratricides  and  Overkills  Using  UASs  in  Network  Centric 

Warfare  Operations,”  Mr.  Stan  Nair,  U.S.  Army  Evaluation  Center 

“Global  Hawk  Concept  to  Combat;  T&E  Lessons  Learned,”  Mr.  Robert  Ettinger, 
Northrop  Grumman  Corporation 

“ATEC  Forward  Operational  Assessment  Team,”  COL  Mark  Mills,  USA,  U.S.  Army 
ATC 

10:00AM 

Morning  Break  &  Networking  in  Display  Area  -  Horizon  Ballroom 

10:30AM 

Display  Area  Closed 

10:30AM 

“Lessons  Learned  in  Fielding  a  UAS  RQ4A  Global  Hawk  in  the  Combat  Theater,” 
Mr.  Sam  McKeehan,  Northrop  Grumman  Corporation 

SESSION  J  -  CONFERENCE  SYNTHESIS:  ACTIONS  AND  TAKEAWAYS 
Chair:  Dr.  Ernest  Seglie,  Science  Advisor,  DOT&E 


1 1 :00AM 

►  Mr.  Christopher  DiPetto,  Deputy  Director,  Systems  &  Software  Engineering, 
OUSD  (A&T) 

►  Mr.  David  Duma,  Principal  Deputy,  ODOT&E 

►  Maj  Gen  George  Harrison,  USAF  (Ret),  Georgia  Tech  Research  Institute 

►  Dr.  James  Streilein,  Technical  Director,  Army  T&E  Command 

►  Mr.  Steven  K.  Whitehead,  Executive  Director,  Navy  OPTEVFOR 

The  discussion  of  this  panel  on  the  overall  thrust  of  the  conference  will  serve  as 
the  basis  of  a  White  Paper  and  an  Overview  article  to  be  published  in  NDIA’s 
National  Defense  Magazine. 

1 1 :55AM 

Closing  Remarks,  Mr.  James  O’Bryon,  Chairman,  NDIA  T&E  Division 

12:00PM 

Conference  Adjourned 

A 
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WALTER  W.  HOLLIS  HONORS  BANQUET 

The  Walter  W.  Hollis  Award  is  presented  annually  in  recognition  of 
lifetime  contributions  and  achievement  in  the  area  of  defense  Test  & 
Evaluation.  The  Award  is  presented  in  the  name  of  Walter  W.  Hollis 
who  is  recognized  for  his  dedicated  and  long-standing  service  in  the 
field  of  Test  &  Evaluation. 

Previous  recipients  include: 

►  Mr.  James  F.  O’Bryon,  Former  DDOT&E /  LFT  (2007) 

►  RADM  Bert  Johnston,  USN  (Ret),  Wyle  Laboratories  (2006) 

►  Hon  Thomas  Christie,  DOT&E,  OSD  (2005) 

►  Dr.  Marion  Williams,  HQ  AFOTEC  (2004) 

►  Mr.  James  Fasig,  Aberdeen  Test  Center  (2003) 

►  Mr.  G.  Thomas  Castino,  Underwriters  Laboratories,  Inc.  (2002) 

►  Hon  Philip  Coyle,  III,  DOT&E,  OSD  (2001) 

►  Mr.  Walter  Hollis,  Department  of  the  Army  (2000) 


TESTER  OF  THE  YEAR  AWARDS  LUNCHEON 

The  awards,  presented  to  outstanding  individuals  in  the  field  of  Test 
&  Evaluation,  offer  OSD  and  each  Military  Service  Test  &  Evaluation 
Department  the  opportunity  to  select  three  award  recipients  for 
recognition  as  the  Tester  of  the  Year  in  specific  categories.  The  three 
categories  recognized  are:  Military,  Civilian  and  Contractor. 


T&E  EXECUTIVE  COMMITTEE 

►  Mr.  Joe  Andrese,  APG 

►  Mr.  Dennis  Bely,  ARL 

►  Dr.  Keith  Bradley,  LLNL 

►  Mr.  Brit  Bray,  DRC  Corporation 

►  RADM  David  Crocker,  USN  (Ret),  CSC 

►  Dr.  Paul  Deitz,  HRED 

►  Mr.  Dick  Dickson,  Tybrin  Corporation 

►  Mr.  Russ  Hauck,  National  Center  for  Simulation 

►  Dr.  Anne  Hillegas,  ARA,  Inc. 

►  Mr.  John  lllgen,  Northrop  Grumman  Corporation 

►  RADM  “Bert”  Johnston,  USN  (Ret),  Wyle  Laboratories 

►  Dr.  Mark  Kiemele,  Air  Academy  Associates 

►  Mr.  Chuck  Larson,  SURVICE  Engineering 

►  Mr.  James  O’Bryon,  The  O’Bryon  Group 

►  Dr.  Ernest  Seglie,  ODOT&E 

►  Mr.  Jack  Sheehan,  FCSCTO 

►  Dr.  Lowell  Tonnessen,  IDA 

►  Dr.  Juan  Vitali,  JPEO-CBD,  OSD 

►  Mr.  Tom  Wissink,  Lockheed  Martin  Corporation 

►  Mr.  Bill  Yeakel,  ORSA  Corporation 
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THANK  YOU  TO  OUR 
PROMOTIONAL  PARTNER 


AIR  ACADEMY  ASSOCIATES 


Air~ 

Academy 

Associates 

Sim  pi  if  Per  fee  t ,  Inn  oval  e 


Air  Academy  Associates  specializes  in  simplifying  the  disciplined  and  meticulous  use  of  powerful  tools  and 
methodologies  such  as  Design  of  Experiments/Multivariate  Testing,  High  Throughput  Testing  (HTT),  and 
Design  for  Six  Sigma.  DOE  provides  more  information  per  data  point  collected  than  any  other  method 
and  is  a  key  to  developing  robust  designs.  HTT,  a  method  for  reducing  the  number  of  test  cases  while  still 
maintaining  adequate  test  coverage,  focuses  on  constructing  the  most  useful  test  combinations  from  a 
large  number  of  variables,  each  with  many  possible  options.  Design  for  Six  Sigma  enables  organizations 
to  decrease  time  to  market  by  25  %  to  40%  which  results  in  overall  cost  savings  of  20%  to  40%  due  to  the 
utilization  of  fewer  resources.  Using  data  and  a  DFSS  scorecard,  we  help  clients  zero  in  on  the  most  likely 
causes  of  failure  rather  than  relying  on  conjecture. 


Our  Competitive  Excellence  Model  provides  our  clients  with  the  plan,  strategy,  and  tactics  -  including  the 
methods,  tools,  and  techniques  -  needed  to  preserve  market  share,  profitability,  and  brand  recognition  and 
to  enable  the  client  organization  to  evolve  by  developing  new  life  cycle  curves  for  new  products/ 
services  or  new  generations  of  products/services.  Our  A3CE  model,  which  has  at  its  foundation  our 
Knowledge  Based  Strategy,  incorporates  Lean,  Six  Sigma,  and  Systematic  Tactical  Innovation  as 
Preservation  activities  and  Design  for  Six  Sigma  and  Systematic  Strategic  Innovation  as  Evolution 
activities. 


Air  Academy  Associates 
Competitive  Excellence  (A3CE)  Model 


Air  Academy  Associates  provides  value  to  our  customers  by: 

►  Educating  leaders  in  our  proven  Knowledge-Based  Business  Strategy  which  emphasizes  Voice  of  the 
Customer,  Strategic  Planning,  and  Systematic  Innovation. 

►  Customizing  our  support  and  materials  based  on  the  client  needs. 

►  Utilizing  mature  and  experienced  consultants  who  are  proficient  at  problem-solving  and  who  excel  as 
classroom  instructors  and  knowledge  transfer  agents. 

►  Incorporating  advanced  methods  and  techniques  such  as  Robust  Design,  Tolerance  Allocation, 
Expected  Value  Analysis,  Axiomatic  Design,  etc.,  into  our  programs  to  optimize  current  performance 
and  to  enhance  design  and  development  phases. 

►  Developing  supplemental  strategic  alliances  to  insure  that  our  clients’  needs  are  met. 

For  additional  information,  please  contact  an  Air  Academy  Associates  Business  Development 

representative  at  1-800-748-1277  or  visit  www.airacad.com. 
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2111  WILSON  BOULEVARD 
SUITE  400 

ARLINGTON,  VA  22201 -3061 
(703)  522-1820 
(703)  522-1885  FAX 
WWW.NDIA.ORG 
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THANK  YOU  FOR  ATTENDING 


WE  LOOK  FORWARD  TO 
SEEING  YOU  NEXT  YEAR  IN 
►  ATLANTIC  CITY  < 
MARCH  2-5, 2009 


Air~ 
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Associates 
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Interoperability  Testing  in  a 
NetCentric  Environment 


Use  of  DoDAF  &  SysML  Profile 
in  the  Flight  Test  Environment 


F.  C.  Alvidrez 

a 

V  CERTIFIED  IKTEBP31SE  ARCHITECT 

X-r  L.#  1  — 


MTS  I  -  Edwards  AFB 


•  Background,  Introduction  &  Acknowledgements 

•  Why  are  we  here? 

-  Transformation  &  the  Growth  of  NetCentric  Warfare 

-  Growth  of  NetCentric  Warfare 

-  System  of  Systems  Engineering 

•  Interoperability  Requirements 

-  Integrated  Architectures 

-  NR-KPPs 

-  DISA&JITC 

•  Approach  to  Testing 

-  Example 

•  Summary 


Background  &  Introduction 


mQD&W  fAruvtk  dtf  r jto:  c/OTA-jr  jrcr 


•  Some  Acknowledgements 


-  DAU 

-  DISA 


-  DoDAF  Working  Group 

-  OMG 


•  Some  Caveats 


-  Not  an  official  USAF  or  USG  presentation 

-  All  Material  is  available  on  line  through  DAU,  DISA 
and  other  public  DoD  Sources 


J/Q&f  W  /JtATHJt  dtf  K  VPCWS.  ,^‘C 


If  you  are  not  interoperable,  you  are... 


Interoperability 


Not  on  the  net 


Not  contributing 
Not  benefiting 


/  V Vi  fjwro 


Source:  USN  War  College  NetCentric  Warfare  Presentation 


•  Increasing  NetCentric  upgrades  and  modifications  since 
9/1 1  (actually  starting  after  Iraq  1) 

•  Link-16,  FAB-T,  B-1B  FIDL,  B-52  CONECT,  B-2A  EHF, 
etc., 

•  Requirement  to  shorten  the  “Kill  Chain”  and  to  increase 
Situational  Awareness 

•  Interoperability  is  about  Integrated  Architectures 

•  DoDAF  is  the  language  of  Integrated  Architectures  in 
DoD 

•  DoDAF  Integrated  Architectures  is  the  blueprint  for 
Interoperability  Test  Requirements 


Interoperability  Concept 
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MILSTAR,  Gapfiller / 
Commercial  Kjj 


Reachback 
to  CONUS 


Theater  ROE 
and  Guidance 


Intel  Broadcasts 


E2-Cj  Joint  STARS, 
Rivet  Joint,  etc. 


UAV 

r 


NIM 

Systems 


Source:  DARPA  J-UCAS  2004 


•  Interoperability  -  Systems  work  at  the 
subsystem  level  but  do  not  work  when  in  an 
operational  environment 

•  Integration  problems  with  legacy  systems  - 
Systems  work  but  interfere  with  legacy  systems 
such  as  IFF,  TACAN  or  Defensive  Avionics 

•  Lack  of  NetCentric  &  Interoperability  Test 
expertise  for  early  Test  Planning  - 


DoD  5000.1  and  DoD  5000.2 

-  Establishes  DoD  acquisition  policy 

-  Requires  select  integrated  architecture  views  at  each  milestone 

-  Content  and  Scope  of  architecture  nroducts  used  is  determined 
by  IMDA  f  PM 

DoD  Architecture  Framework  (DQDAF)  Document 

-  P  rcjvi  d  e  s  basis  for  d  eve  loping  sta  ndard  ized  a  rch  itectu  re  vi  ews 
and  products  required  by  DOD  CJCS,  and  SECNAV  policy 
documents 

CJCSI  31 70.01  E 

-  Requires  the  development  of  integrated  architecture  products 
for  supporting  acquisition  documentations: 

■  Joint  equability  Integration  and  Developn1  ent  System  (JCIOS) 

■  Ca  cabiity  gap  and  redundancy  analysis 

CJCSI  S212.01C 

-  Requires  architecture  products  be  used  in  the  J-6 
interoperability  and  supportability  certification  process 

-  Specifies  which  architecture  products  are  required  for  the  ICD, 
ODD,  and  CPD 

SECNAV  INST  5000.2C 

-  Establishes  DON  acquisition  policy 

-  Directs  the  PEO  I  PM  to  develop  mission  integrated 
architectures  in  suoport  of  the  CDD/CPD  process 

-  Directs  ASN  (RD&A}  CHENG  to  assist  PMs  in  the  development 
of  operational  and  system  architectural  views. 


DoD  4630  -  All  IT  &  NSS  must  be 

the  test  results  evaluated  and 


before  fielding,  and 
by  DISA  (JITC) 


DoDAF 

:&r 

S/lar 

idatory 

Interoperability 
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DoD  Dir.  4630.5  Interoperability  and  Supportability  of 
Information  (IT)  and  National  Security  Systems  (NSS) 

DoD  5000  Series  -  Defensive  Acquisition  System 

Joint  Capabilities  Integration  and  Development  System 

(JCIDS)CJCS  3710  Series 

CJCSI  3180.01  Joint  Requirements  Oversight  Council 
(JROC)  Programmatic  Process  for  Joint  Experimentation 
and  Joint  Resource  Change  Recommendations 

Others 


How  do  you  ensure  interoperability?  -  Compliant 

DoDAF  Architectures 


DoD  Architecture  Framework 
Purpose  &  Scope 


Provide  guidance  on  describing 
architectures 

-  Warfighting  Operations 

-  Business  Operations  &  Processes 


Provides  guidance,  rules  &  product 
descriptions  for  developing  and  presenting 
“Integrated”  architectures 

Defines  three  views  /*  | " ' "  |  V 

-  Operational  View  ^  | 


-  System  View 


Sy&lBms 

Vlaw 


Rh Uk  a  ysta  m  c  axl  C  w  srtr;  I  w 
Id  OparaUsml  Naodc 


RkliivJ 


Technical  Standante 
View 


ibKltmlaifci 

Cormrantfetw 


-  Technical  Standards 


Figure  3-2.  Fundamental  Linkages  Among  1h*  v'le^a 


Systems 

View 


Relates  Systems  and  Characteristics 
to  Operational  Needs 


1  1 


Operational 

View 


Identifies  What  Needs  to  be 
Accomplished  and  Who  Does  It 


I 


Specific  System  Capabilities 
Required  to  Satisfy 
Information  Exchanges 


Technical  Standards  Criteria 
Governing  Interoperable  '— 

Implementation/Procurement 
of  the  Selected  System  Capabilities 


Prescribes  Standards  and 
Conventions 


Figure  ES-1.  Linkages  Among  Views 


Technical 

View 


The 

Operational  View 
describes  and 
interrelates  the 
operational  elements, 
tasks  and  activities,  and 
information  flows  required  to 
accomplish  mission  operations. 

The 

systems  View 
describes  and 
interrelates  the 
existing  or  postulated 
technologies,  systems,  and 
other  resources  intended  to 
support  the  operational  requirements] 


The 

Technical  View 
describes  the  profile  of  rules, 
standards,  and  conventions 
governing  systems  implementation. 


Source:  DoD  AF  1 .0  Draft 
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UuD  Architecture  Framework. 
Version  1.5 


_ 


2  AVs 
9  OVs 
16  SVs 
2  TVs 


Table  2-1:  List  of  Products 


Apphcable 

Framework  Product  name 

Net  Centric 

General  Description 

All  View 

AV-1 

Overvie*  end  Summery  wlomteHcn 

✓ 

Scape,  purpose  nended  users,  ermrormonl  depicted, 
analyncal  fndrgs 

AH  View 

AV-2 

integrated  Diclcna-y 

S 

A-cnaesu-e  errs  rep.»liofy  with  detritions  of  ell  te-rre 

Operational 

OV-1 

Hiyi  Level  Oseraccral  Concept  Graphic 

Y 

Operational 

OV-2 

Operational  Nc«te  Conoeclwity 

Y 

Ooerstorel  none*,  conrectiwty,  and  information 

Operational 

OV-3 

Operational  Information  Exchange  Matrix 

Y 

Infcmaton  exchanged  between  nodes  arc  lie  relevant 
attnbutaeofttttexhange 

Operation** 

OV-4 

Organi&ttcrva  Rwaocrships  Chart 

Y 

organ/ationa 
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Sendee*  interface  Desertion 

Y 

Identificsncr  &  systems  nodes,  systems,  system  terns. 
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wwce  llama  and  ther  -elatec  common  cations  toy- 

Sjpn, 

« 

Systems  Systems  Mattix 

Sennces-Sytaeme  Matrix 

Y 

Rel8tk>n*npe  among  kyelems  end  services  r  s  gven 
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Phyaksa  imp4emertar.cr  of  me  Logical  Data  Mote 
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Technical 
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TV-2 
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a  sat  of  time  frames 
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DAU  1ST 


T&E  Step  3 
Conduct  Test 
Activities 


Joint  Interoperability 
Directives  &  Instructions 


DODD  4630.5 


IT  and  NSS  Interoperability 
shall  be  verified  early, 
and  with  sufficient 
frequency  throughout  a 
system's  life  ... 


DODD  8500.1 


ALL  IT  must  be 
evaluated  and 
validated  for  IA 


POD  4630/5000 

Interoperability  is 
"the  ability  to  provide  and  accept  data , 
information,  materiel,  and  services  ... 
...includes  both  the  technical  exchange  of 
information  and  the  end-to-end 
operational  effectiveness  of 
that  exchange,  as  required  for 
mission  accomplishment 


UNCLASSIFIED 


CJCSI  6212.01  D 


All  IT  and  NSS  must  be 
evaluated  and  certified  for 
Joint  interoperability  by  PISA 

(JITC), 


series 


DO  PI  4630.8 


All  IT  and  NSS  ...  must  be 
tested  for  Interoperability 
before  fielding  ...  and  certified 

by  DJSAfJITC}. 


CJCSI  3170.01  E 


Establishes  JCIDS 
w i  NR-KPP  for 
CRD,  CDD 
and  CRD. 


For  IT  systems,  Including  NSS, 
...  JITC  shall  provide  system 
Interoperability  test 
certification  memoranda ... 
throughout  the  system  life¬ 
cycle  and 

regardless  of  ACAT. 


i^/wtfcmeiairSQieiJiMs  ixc 


ns/v^ 


UNCLASSIFIED 

Interoperability 
Certification  Timeline 


Draft  CDD  ■  CertfflBd  CDD.  Initial  ISP 
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Sys  fern  Deve/opmenr  &  flefnonsif.rafro.n  Phase 


Low-Rate  initial 
Production 
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£2E  Info  Exchange 
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itional,  System 
Ttthni^Vi«w  integration 


CO  I  and  Enterprise 
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KIP  Integration 


DIACAP  A^vity  3 
CompEl  a  nee 


Mission  Effectiveness 
E2E  Into  Exc  h* 
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Integrated 

Comporwnf* 


Opanttefial,  S: 
View  Oi 


Interoperability 
Tests  in  Place 


|ry 


MSS  >'  IT 


Tfrthnle*l 
Fstration 


COl  and  Enterprise 
Datef  Service  Compliance 


KIP  Compliance 


Supports  Critical 
OpereMontl  Issues 
ft,  Measures 
of  Effectiveness 


DIACAP  Activity  3 
Compliance 


Assess 

Technology 


Demonstrate  System 
Integration  and  Interoperability 


Achieve  Operational 
Capability 
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DoDAF  Integrated  Architectures 


Initial  SE 
Requirements 


Full  Interoperability 
Demonstration 
(JITC  Certification) 


Increasing  product  maturity  and  operational  realism 
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Technical  Standard*  VlfW 
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to  CONUS 


Theater  ROE 
and  Guidance 


V' 

Intel  Broadcasts 


E2-C,  Joint  STARS, 
Rivet  Joint,  etc. 


UAV 
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^  A  __  Commercial  Ku 


NIM 

Systems 


Source:  DARPA  J-UCAS  2004 


DAU  NR-KPPs  Web  Course  (CLM  029) 

DAU  Fundamentals  of  Test  &  Evaluation 
Web  Course  (TST  102) 

DISA/ JITC 

(http://www.disa.mil/index.html)  / 
(http://jitc.fhu.disa.mil/jitc_dri/jitc.html) 

DARS  -  DoD  Architecture  Repository  1 
System 

(https://dars1.army.mil/IER/welcome.jsp) 


DcfrnKAtqwtoon  Unhrtwty 


•  Interoperability  modifications  and  upgrades  are 
increasing 

•  Interoperability  is  System  of  Systems  Engineering 

•  Interoperability  Testing  has  special  considerations 
(other  nodes  and  players) 

•  Can  use  simulations  as  well  as  hardware  in  the 
loop.  Will  require  flight  demonstrations  in  a 
representative  environment  (JITC  Certification) 

•  DoDAF  Architectures  are  required  in  all  phases  of 
Interoperability  Testing  (TES  -  TEMP  -  Flight  Test 
Plan  -  JTIC  Certification) 

•  Recommendation  -  Have  a  DoDAF  Architect  on 
the  TRWG  (Test  Requirements  Working  Group) 
early 


IN 


Robotics 


A>j 

>tics 


FIRST 


Motivating  Young  Americans  to 
Pursue  Robotics  Technology 
NDIA  24th  T&E  Conference  *  Feb  2008 


Motivating  Young 
Americans  to  Pursue 


Robotics  Technology 

With  Support  From 

■  University  of  Maryland  Baltimore  County 

■  SURVICE  Engineering 

■  Maryland  State  Senator  Nancy  Jacobs 

■  NJ  FTC 

■  iCNRG  FLL 


Silverpop,  Inc. 

Creative  Kids  at  Home 
Leavitt’s  Freight  Service 
Home  Education  Resources 
Pace  Academy 
Custom  Direct 
Nehemiah  Communications 
The  Better  Hour 
Broadway  Appraisals 
Madonna  Veterinary  Clinic 
Consolidated  Printing 
Enktesis,  LLC 
Numerous  FIRST  Teams 


Marco  Ciavolino 

Founding  Mentor  for 
TechBrick 

Marketing/Communications 
&  Technology  Consultant 

*enktesis,  LLC 

Forest  Hill,  Maryland 
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Why  Would  You  Be  Interested  in 
Teaching  Young  People  Robotics? 

Because  we  need  bright,  innovative,  young  engineers 
to  create  and  maintain  our  future  technologies. 


■  Good  engineers  and  technologists  are  raised  before  they  are 
taught.  The  penchant  for  mechanical  and  conceptual 
disciplines  comes  from  a  lifetime  of  involvement. 

■  For  the  past  five  years  we  have  coached  robotics  teams 
under  programs  offered  by  FIRST. 

FIRST  programs  grow  engineers  and  technologists. 
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Why  Are  You  Hearing  About  This  at  NDIA? 


The  Programs/Products  You  Support  Will  Require 
Young,  Smart,  Capable  Engineers 

■  We  will  show  a  program  with  more  than  200,000  participants 
world-wide. 

■  We  will  show  you  a  program  that  doubles  or  triples  a 
student’s  interest  in  science  and  technology. 

■  We  will  show  you  a  program  that  will  bring  to  your  future 
projects  the  talent  you  need. 

■  Some  NDIA  divisions  are  already  working  with  FIRST,  we 
need  the  rest  of  you  to  consider  involvement. 


The  programs  are  offered  through  US  FIRST... 
What  is  US  FIRST? 


Marco  Ciavolino  *  Marco@TechBrick.com  *  www.TechBrick.com  *  www.USFirst.org 


IN 


Robotics 


A>j 

>tics 


FIRST 


Motivating  Young  Americans  to 
Pursue  Robotics  Technology 
NDIA  24th  T&E  Conference  *  Feb  2008 


What  is  US  FIRST? 


Founded  by  Dean  Kamen 

■  FIRST  was  founded  in  1989. 

■  To  inspire  young  people's  interest 
and  participation  in  science  and 
technology. 

■  Provides  accessible,  innovative 
programs  that  motivate  young 
people  to  pursue  education  and 
career  opportunities  in  science, 
technology,  engineering,  and  math, 
while  building  self-confidence, 
knowledge,  and  life  skills. 
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For 

Inspiration  and 
Recognition  of 
Science  and 
Technology 


FIRST 


This  is  without  a  doubt 
one  of  the  most 
engaging  and 
challenging  programs 

available. 
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What  are  the  Programs? 


J  FLL 

JR,  FIRST  LEGO*  League 


o  Asfr 

FLL  FTC  FRC 

FIRST  LEGO  League  FIRST  Tech  Challenge  FIRST  Robotics  Competition 


For  ages  5-8 

Ages  7-15 

Jr.  High  -  HS 

High  School 

Uses  the  annual 
theme  for  project- 
based  work. 

Uses  annual  themes 
to  engage  young 
students  in  applied 
research. 

Uses  a  mechanical  challenge  combined  with 
real-world  teamwork  and  cooperative 
efforts. 

Based  on  standard 
science  project 
materials 

Based  on  LEGO 
Mindstorms  NXT  or 
RCX  Robotics 
Systems. 

Based  on  an  advanced  robotics  system. 

Up  to  6 

Up  to  10 

Up  to  10 

As  many  as  needed. 
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What  are  the  Programs? 

JFLL:  Junior  FIRST  LEGO  League 

■  Based  on  LEGO  Education  Kits. 

■  2-6  Students. 

■  Thematic  Challenges. 


JFLL 

JR.  FIRST  LEGO*  League 
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What  are  the  Programs? 

FLL:  FIRST  LEGO  League 

■  Based  on  LEGO  Mindstorms  NXT  or  RCX 
Robotics  Systems. 

■  4-10  Students. 


Thematic  Challenges:  4  Parts 


Ms 

PAMIRS 
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What  are  the  Programs? 

FTC:  FIRST  Tech  Challenge 

■  Based  on  advanced  robotics  systems. 

■  2-10  Students. 

■  Uses  a  mechanical  challenge  combined  with 
real-world  teamwork  and  cooperative  efforts. 


FTC 

FIRST  Tech  Challenge 


QUAD 
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What  are  the  Programs? 

FRC:  FIRST  Robotics  Competition 

■  Based  on  advanced  robotics  system. 

■  10-60  Students. 

■  Uses  a  mechanical  challenge  combined 
with  real-world  teamwork  and  cooperative 
efforts  with  mentors  and  sponsors. 
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What  are  the  Programs? 


International  Competition 

■  10,000  students 

■  6000  mentors 

■  1000’s  of  volunteers 

■  3  Programs 

✓  FLL,  FTC,  FRC 
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What  are  the  Results  of  This  Work? 

The  Numbers 

■  Steady  Growth  and  Involvement 


J FLL  FLL 

JR  RUST  LEGO*  League  FIRST  LEGO  League 


As* 

FTC  FRC 

FIRST  Tech  Challenge  FIRST  Robotics  Competition 


3  Years 
1000  Teams 


10  Years 
10,000  Teams 


4  Years 
1200  Teams 


14  Years 
1400  Teams 
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What  are  the  Results  of  This  Work? 


Colleges:  More  than  100  Colleges  Offer  Scholarships 


Colorado  State  University-Pueblo 
Arizona  State  University 
ASME 

Boston  University 

Clarkson  University 

Cleveland  Institute  of  Art 

College  for  Creative  Studies 

Colorado  Technical  University-Denver 

Daniel  Webster  College 

DeVry  University 

Drexel  University 

Eastern  Michigan  University 

Embry-Riddle  Aeronautical  University 

Fairleigh  Dickinson  University 

Ferris  State  University 

Florida  Atlantic  University 

Florida  Institute  of  Technology 

Georgia  Institute  of  Technology 

Grand  Valley  State  University 

Hampshire  College 

Henry  Ford  Community  College 

Hofstra  University 

Idaho  State  University 

Illinois  Institute  of  Technology 

ITT  Technical  Institute  (combined) 

Kansas  State  University 


Lake  Superior  State  University 
Lawrence  Technological  University 
Marquette  University 
Massachusetts  Institute  of  Technology 
Mercedes-Benz  USA  LLC 
Michigan  State  University 
Michigan  Technological  University 
New  Hampshire  Technical  Institute 
New  Jersey  Institute  of  Technology 
Northeastern  University 
Ohio  State  University 
Olin  College  of  Engineering 
Oregon  State  University 
Pennsylvania  College  of  Technology 
Phil  Clancy  Scholarship 
Polytechnic  University 
Purdue  University/Delphi  Corporation 
Raytheon  Company 
Rensselaer  Polytechnic  Institute 
Rochester  Institute  of  Technology 
Schoolcraft  College 
Society  of  Women  Engineers 
Southern  California  Regional  Robotics 
Forum 

Spring  Arbor  University 
Temple  University 


University  of  Arkansas-Little  Rock 
University  of  California-Davis 
University  of  Central  Florida 
University  of  Delaware 
University  of  Denver 
University  of  Hartford 
University  of  Illinois-Chicago 
University  of  Kansas 
University  of  Massachusetts  Lowell 
University  of  Michigan/Delphi 
University  of  Minnesota 
University  of  Missouri-Kansas  City 
University  of  Nebraska-Lincoln 
University  of  New  Hampshire 
University  of  Rochester 
University  of  South  Carolina 
University  of  Southern  California 
University  of  Toronto 
University  of  Waterloo 
Virginia  Commonwealth  University 
Washington  State  University 
Washtenaw  Community  College 
Wayne  State  University 
Wisconsin  Lutheran  College 
Worcester  Polytechnic  Institute 
Kettering  University 
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What  are  the  Results  of  This  Work? 


Over  $9.5  million  in  scholarship  funds 
available  to  FIRST  participants 


MICHIGAN  STATE 


UNIVERSITY 


Drexel 

FKI^lJUlirt 


BOSTON 


UNIVERSITY 


UnlverftJty  of 

Waterloo 


MichiganEngineedng 


UCF 


Stands  For  Opportunity 


DeVry  ^ 

University 

Georgia 
_  Tech 


UtnVMSmr  £#  rjaxPOftrtW, 


I  UNIVERSITY  OF  DENVER 


10,000,000 
9,000,000 
8,000,000 
7,000,000 
6,000,000 
5,000,000 
4,000,000 
3,000,000 
2,000,000 
1,000,000 
0 

^  ^  ^  ^15 
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What  are  the  Results  of  This  Work  For  Our  Nation? 


The  Alumni 

■  FIRST  Students  vs.  Comparison  Group: 

S  Seek  Education  in  Science  &  Technology. 

S  Twice  as  likely  to  major  in  science  or  engineering. 

S  More  than  three  times  as  likely  to  major  specifically  in 
engineering. 


100% 

80% 

60% 

40% 

20% 

0% 


■  FRC  Alumni  ■  Matched  Comparison  Group 


Attend  College  Aspire  to  Post-  Major  in  Science  Major  Specifically 

Graduate  Degree  or  Engineering  in  Engineering 
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What  are  the  Results  of  This  Work  For  Our  Nation? 


The  Alumni 

■  Earn  Career  Opportunities: 

S  Almost  ten  times  more  likely  to  have  an  internship. 

■  Expect  to  Pursue  Science  &  Technology  Careers: 

S  More  than  twice  as  likely  to  pursue  S&T  career  . 

s  Nearly  four  times  as  likely  to  pursue  career  specifically  in 
engineering. 


Internship/Co-op  Job  Expect  to  Have  Pursue  Career 

During  FreshmanYear  S&T-relaced  Career  in  Engineering 
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Real-World  Results:  TechBrick  Robotics 


Our  History 

■  TechBrick  was  formed  2004 

■  First  competition  in  2005 

■  Growth  from  20-120  members 

■  Monthly  activities 

■  30  team  participants  in  2007-08 

■f  JFLL,  FLL,  &  FTC 

■  Interest  continues  to  grow 

■  Organized  Maryland  JFLL 
Tournament 

■  Regional  seminars  on  starting 
a  team 

■  First  high  school  judges  for  FLL 
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Real-World  Results:  TechBrick  Robotics 

Monthly  Meetings 

■  More  that  120  children 
registered 

■  Builds  interest  for  teams 

■  Creative  exercises 

S  Animation 
v'  Documentation  Tasks 
S  Boat  building 
S  Bridge  Building 
s  Guest  Speakers 
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Real-World  Results:  TechBrick  Robotics 


Competition  Teams:  Innovative  Solutions 

■  Light  Sensor:  Improved  by  1200%  (FLL)  ry 

S  Using  a  ‘Spider  Man  Magnifying  Glass’  gj 

■  Sideways  Stepping  Bot  (FLL) 

S  Drive  forward,  step  right  and  left  H 

■  Torque/RPM  Discoveries  (FLL) 

s  Design  to  specifications  exercise  H 

-  Ball  and  Bar  Lifting  Bot  (FTC) 

s  Capture  the  primary  object 

■  Ring  Lifting  Automation  Device  (FTC) 

^  Pickup  rings  with  materials  | 

handling  device  L_ 


Marco  Ciavolino 
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Real-World  Results:  TechBrick 


State  Competitions 

■  UMBC  (University  of  Maryland,  Baltimore  County) 

■  TCNJ  (The  College  of  New  Jersey) 

■  UD  (University  of  DE) 

■  CSM  (College  of  Southern  Maryland) 
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Real-World  Results:  TechBrick  Robotics 
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Real-World  Results:  TechBrick  Robotics 


A  Robust  Website  with  Useful 


Resources 


■  Team  Tips 

■  Worksheets 

■  Projects  for  group  work 

■  Photos  and  guidelines 

■  Global  traffic 


inada 


North 

Atlantic 

Ocean 


South 

Pacific 

Ocean 


HOT  NEW  ITEMS! 
Worksheets  for  the 


Power  Puzzle  Challenge 


Get  TechBrick  Clothing, 


|%  h  ftp :  //www .  techbrick . 


^  Welcome  to  TechBrick  2007-08  First  Lego  League  and 


ALERT:  Marco  Ciavoiino  to  Address 
National  Defense  Industrial  Association 
US  FIRST  and  Student  Robotics  Training. 


2007-08  Site 


Click  here  for  more  information. 

•ilfCHBRICfc 


2007-08  Teams: 
FTC  Team  7  *  FLL  Team  10 
JFLL1  *  JFLL2 

Welcome  to  TechBrick  the  Web  Site  of  the  New  Covenant 
Christian  Academy  Home  School  Robotics  Club  in  Abingdon. 

Maryland,  USA 


TechBrick.com  2007-2008 


Welcome  to  TechBrick 
for  2007-08 


The  Home  of  the  New  Covenant  Christian 
Academy  Robotics  Club 


What's  the  latest? 


We  took  some  of  our  team 
members  to  a  local  home 
school  meeting  to 
demonstrate  our  three 
programs,  JFLL,  FLL,  and 
FTC. 


We  need  your  help.  Please 
read  this:  TechBrick  has  been 
invited  to  address  the  Address 
National  Defense  Industrial 
Association  on  US  FIRST  and 
Student  Robotics  Training.  This 
past  season  exhausted  all  our 
team's  financial  resources.  I'm 
asking  each  team  to  consider 
contributing  just  $20  or  more  to 
the  effort.  Click  here  for  more 
information. 


Along  the  way  we  met  R2D2,  too, 
and  gave  him  a  TechBrick  hat.... 


Check 


^  I  NEW!  TechBrick 

rTirffiC7  <  1 1 11 c  Tr  Wins  the 

'Amaze'  Award  at 
M  A  fi|the  New  Jersey 
M  1 t&'W  FTC  Tournament 

rLJ  llbr  Ion  February  9  at 
the  The  College  of  New  Jersey,  see  photos 
and  video  of  the  unique  lifting  device  in  action.  Check  out  the 
photos  and  video 


Interested  in  starting  a  LEGO 
Robotics  club?  Consider  registering 
your  interest  in  our  'How  to  Start  a  US  FIRST 
Team'  seminar  this  spring.  Click  here  for 
more  information. 


Check  out  highlights  of  the  past  four  years! 


NEW!  FLL,  JFLL  &  FTC 
Continue  Their  Work 


Tor.hBrirk  fpatiirpri  in  local 
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OQ 

rTSeien 

F  \  LEARN  E: 


lienee  is  Fun! 

LEARN  EXPLORE  AND  PLAY 


Real-World  Results:  TechBrick  Robotics 


Community  Service 


Library  Programs 


Local  Presentations  to  Sponsors 


SURVICE  Engineering  Corporation 
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Real-World  Results:  TechBrick  Robotics 


;«te$cpoo| 


v.  m 


Robotics  team  heads  for  international  competition 

Fiw  Harford  hqme-seboolers  hmld  bfl-ts  Mr^ 

HK  T&i  JEwT'tL 


5  tv-i+tytumW} 

wmmm 


Adopted  Team  Members 


Media  and  Fun 


HARFORD  COUNTY'S  PARENTING  MAGAZINE  Celery  2O0tt  *  FKEt 


Featured 
in  Media 
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Real-World  Results:  TechBrick  Robotics 


Focused  Children:  Good  for  America... 


Of  the  50+  children  on  teams  over  the  past  five  years  at  least 
half  have  been  encouraged  to  pursue  math,  science,  and 
engineering  studies. 

Numerous  innovations  in  research,  design,  and  execution. 
Lessons  learned  in... 

S  Project  Management 
S  Processes 
S  Teamwork 
S  Implementation 
S  Competition 
S  Design 
S  Research 
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Real  World  Results:  Around  The  World 


Our  Experiences  Have  Been  Repeated  Around  the 
World 


Thousands  of  schools  and 
organizations  have  done 
exactly  the  same  work. 

Young  engineers  are  now 
in  the  workplace  or  on 
their  way  to  productive, 
effective  careers. 
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TechBrick  Robotics:  Our  Dream 

A  Regional  Robotics  Center 


floor  plan 


the  pits  and  large-scale  workshop 


the  arena 
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TechBrick  Robotics:  Our  Dream 


The  First  National  Robotics  Center 

■  A  multi-use  facility 
S  Ongoing  classes 

s  Regional  and  national  robotics  competitions 
S  Technology  tradeshows 
S  Technology  meetings  and  conferences 
s  Parties  and  events  (themed) 

S  Educator  training  (CEU) 
s  College  Internships 

s  Outdoor  space  for  large  vehicles  and  displays 
s  Robotics  ‘mini-proving  grounds’ 
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How  Can  You  Help? 


These  Are  Some  Of  The  Attending  Companies  That  Are 
Already  Involved  Nationally  With  FIRST... 


BAE  Systems 

The  Boeing  Company 

Air  Force  Operational  Test  and  Evaluation 

Center 

Northrop  Grumman  Corporation 
General  Dynamics  Corporation 
Lockheed  Martin 

Naval  Air  Warfare  Center  Weapons  Division 


Aberdeen  Test  Center 
Edwards  Air  Force  Base 
GA  Tech  Research  Institut 
Honeywell  International,  Inc. 
Lockheed  Martin 
Raytheon  Company 
US  Army 
USAF 


Let’s  Take  a  Look  at  The  Benefits  of  Getting  Involved... 
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How  Can  You  Help? 


Consider  the  Opportunities 

Supply  equipment/parts. 

Offer  scholarships. 

Provide  facilities  for  teams  and  events. 
Assign  mentors,  volunteers,  consultants. 


Create  internship  opportunities. 
Financial  support. 
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How  Can  You  Help? 


Consider  the  Benefits  To  You  and  To  Our  Nation 

■  Strengthens  public  relations  and  community  relations. 

■  Builds  national  technological  literacy. 

■  Creates  an  incubator  for  interns  and  future  employees. 

■  Motivates  volunteer  opportunities  for  employees. 

■  Applied  professional  development  for  employees. 
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How  Can  You  Help? 

Find  a  US  FIRST  team  in  your  region  and  get 
involved: 

■  Visit  the  US  FIRST  website  at 
www.USFIRST.org  to  get  a  full  understanding 
of  the  program. 

■  Be  a  champion  for  participation  within  your 
company  or  association. 

■  Start  a  FIRST  program  in  your  child’s  school. 

■  If  you’re  from  the  Aberdeen/Edgewood  Area 

S  Consider  assisting  TechBrick  with  a  corporate 
sponsorship.  Visit  www.techbrick.com/support 
for  more  information. 


IchBricK 


Main  Portal 


2007-08  Teams: 
FTC  Team  7  *  FLL  Team  10 
JFLL1  *  JFLL2 


mm- 


m  Paqe  1 2007-OtJ  Site  I  Search  I  Sit 


^  Welcome  to  TechBrick: 

*i*l  s'*  Main  Portal 

The  Home  of  the  New  Covenant  Christian 
Academy  Robotics  Club 


HOT  NEW  ITEMS! 
Worksheets  for  the 
Power  Puzzle  Challenge  I 


We  need  your  help.  Please  read  We  took  *om®  of  our  ,e 
this:  TechBrick  has  been  Invited  to  members  to  a  local  hor 
address  the  Address  National  sch°o1  meeting  to  dem 

Defense  Industrial  Association  on  0t"  ert*  program8’ JF 
US  FIRST  and  Student  Robotics  andFTC. 

Training.  This  past  season  _ 

exhausted  all  our  team's  financial  R2D2,° 
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Who  to  Contact... 

■  Contact  Marco  Ciavolino  for  information  about  The  National 
Robotics  Center  or  TechBrick 
(marco@techbrick.com,  41 0-838-8264) 


■  For  information  about  involvement  in  FIRST  nationally, 
contact  Cindy  Randall  for  information. 
(crandall@usfirst.org,  603-666-3906  x403) 
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A  Simple  Way  to  Assist  Today... 


The  iRobot  Roomba  is  clearly  the  first  successful  home  robot. 

Stop  by  our  booth  for  a  chance  to  win  a  Roomba... 

Keep  your  home  or  office  ‘Roomba’  clean... 


Make  a  suggested  contribution 
for  one  or  more  tickets.  Join  us 
at  the  awards  luncheon  on 
Wednesday  for  the  drawing. 


Our  thanks  to  iRobot  for 
providing  the  Roomba 


Home  Robots 


le  Robots  >  Robots  >  About  Roomba  500 


0  iRobot  Corporation:  About  Roomba  500  -  Windows  Internet  Explorer 


I  ^  h  ttp :  //www .  irobot .  com/sp .  cfm  ?pageid =122 

A  <£?  Robot  Corporation:  About  Roomba  500 


a Robot 


Choose  your  robot  type: 


Floor  Washing 
Shop  Sweeping 
Pool  Cleaning 
Gutter  Cleaning 
Virtual  Visiting 


Roomba  cleans  routinely 
so  you  don't  nave  to! 


iRobot®  Roomba®  Vacuum  Cleaning  Robots 


Maintain  cleaner  floors  every  day! 

Life  happens  in  busy  homes.  Stay  ahead  of  messy  floors  with  the  touch  of  a  button! 
Smart  and  efficient  iRobot  Roomba  lifts  an  amazing  amount  of  dirt,  dust,  pet  hair, 
cat  litter,  crumbs  and  other  debris  from  your  carpets  and  hard  floors. 

•  Powerful  suction  and  rotating  brushes 

- m  a  •  Automatically  navigates  for  best  cleaning  coverage 

•  Cleans  under  and  around  furniture,  into  corners  and  along  wall  edges 

•  Self-adjusts  from  carpets  to  hard  floors  and  back  again 

•  Automatically  avoids  stairs,  drop-offs  and  off-limit  areas 

•  Simple  to  use —  just  press  the  Clean  button  and  Roomba  does  the  rest 


Introducing  the  NEW  generation 


L 

of  iRobot®  Roomba1  Innovation!  ► 

UL 

1 

HI©  internet 

\  100 

%  T  /A 
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Warfare  Systems  Engineering  Challenges  and  ^eSftjand 


Evaluation  Approaches 

Presented  to: 

24th  Annual  National  Test  & 
Evaluation  Conference  Palm 
Springs,  CA  February  25-28,  2008 

Presented  by: 

Rob  Connerney,  Deputy  NAVSEA 
Warfare  Center  T&E  Executive 


40 'J  rd'd2-2'\v'\ 

&*jfr  I  JjjjJ 


Outline  of  Presentation: 

DoN  T&E  Challenges  &  Strategies 
NAVSEA  Core  T&E  Functions 
NAVSEA  T&E  Initiatives 
(Integration,  Interoperability,  M&S, 
Undersea  Tracking  Ranges  and  T&E 
capabilities,  Human  Capital  Strategy) 


“Approved  for  Public  Release;  Distribution  is  Unlimited” 


SEA  05 


Naval  Systems  Engineering  Directorate 


Challenges 

•The  growing  need  for  the  large  scale 
joint  and  dynamic  test  environment  of 
the  future,  e.g.  Net  Centric  Warfare, 
FoS/S,  coalition  operations. 

•  The  increasing  technological 
complexity  and  tightly  integrated 
interoperability  across  weapons 
systems. 

•  Current  information  and  accounting 
systems  do  not  provide  adequate 
visibility  of  T&E  events  and  costs  to 
maintain  and  improve  process  and 
capability. 


Naval  T&E  Challenges: 
Improving  the  T&E  Process 

Strategies 

•  The  Joint  Capability  Integration 
Development  Requirement  and  the 
budget  process  needs  to  be  better 
synchronized  to  reduce  unanticipated 
cost  changes. 

•  The  use  of  combined  Integrated 
DT/OT  is  one  of  the  successful 
“innovative”  T&E  approaches  by  Navy 
program  managers  to  cut  testing  costs. 

•  Increased  use  of  M&S  can  reduce 
Acquisition  programs  costs  and 
improve  the  value  of  DT  and  OT  data. 

•  The  return  of  M&S  across  programs 
can  be  enhanced  through  investments 
in  common  infrastructure,  policies,  and 
standards  and  reuse. 
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Navy  T&E  Initiatives 


SEA  05 


Naval  Systems  Engineering  Directorate 


Navy  T&E  Board  of  Directors 

-  Established  by  DASN/RDA 

-  RADM  Landay  (co-chair)  and  COTF  (co-chair),  OPNAV  N091  (Exec  Sec), 
PEOs,  N1,  N4,  N6,  N8,  SYSCOMs,  MCOTEA  and  CFFC 

-  Strategic  Priorities 

•  Establish  governance 

•  Inventory  the  domain 

•  Define  metrics 

•  Create  value  by  optimizing  resource  utilization 

•  N43  Range  Contribution  to  Readiness  Effort 

•  PEO  T&E  Forum 

-  Coordinates  T&E  actions  across  PEOs 

-  Identify  opportunities  for  efficiencies 

•  PEO  Ships  T&E  Working  Group 

•  NAVSEA  Warfare  Center  T&E  Working  Group 

-  Representatives  from  each  WC  division 

-  Forum  for: 

•  WC  leadership  in  joint  initiatives 

•  Knowledge  sharing,  synergy/efficiency  initiatives 

•  Recommendations  for  T&E  policy  formulation 

•  Coordination  of  NAVSEA  T&E  action  items 

Renewed  focus  and  attention  on  T&E 


DoN  Integrated  Testing  Strategies 


•FY07  Nat'l  Defense  Authorization  Act, 

Sect  231.  directed  USD  and  DOT&E  to 

review  POD  policies  and  practices  on 

T&E. 

•Integrated  testing  major  theme 

-  Goal:  early  detection  &  correction  of 

program  deficiencies 

•  Navy  Proposed  Draft  DoD  IT  definition: 
Integrated  testing  is  the  collaborative 
planning  and  collaborative  execution  of 
test  phases  and  events  to  provide  data  in 
support  of  independent  analysis, 
evaluation  and  reporting  by  all 
stakeholders  (government,  contractor, 
operational  test  communities). 


•PM  identifies  program  as 
“Integrated  Test”  program 

•  Establishes  a  test  team  to 
collaboratively  create  and  manage 
the  TEMP 

•  Identify  test  parameters,  data,  and 
resources  required  for  development 
of  DTI  OT  test  plans,  and 
certifications,  to  optimize  test  data 
collection  while  minimizing  test 
resource  required. 

•  MDA  provide  formal  direction 
establishing  the  Test  Team  in  the 
program's  first  ADM 

•  Contractor  full  participation  in  the 
IT  planning  and  execution  included 
in  RFP  and  contract. 
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NAVSEA  T&E  Initiatives  In  Support  of 

DoN  Directives 


Naval  Systems  Engineering  Directorate 


•  Implement  Integrated  T&E  Strategies 

-  Combined  DT/OT 

-  Synergy  DT/OT  with  Fleet  Training  Events 

*  Implement  Interoperability  T&E 

-  Distributed  Engineering  Plant 

-  Joint  T&E  Distributed  Engineering  Plant 

-  Coalition  T&E 

-  Integral  Fire  07 

•  Promote  Consolidated  T&E  Capability 

-  Undersea  Tracking  Range  Collaboration  and  Roadmap  Development 

-  Synthetic  Environments  to  Enhance  ASW  Operational  Effectiveness 

-  Test  Assets  &  M&S 

-  T&E  Human  Capital  Strategy 

*  Establish  Strategies  for  Transformational  T&E 

-  Establish  Rapid  Response  CREW  T&E 

-  Establish  M&S  Accreditation  Process 

-  Establish  Open  Architecture  T&E 
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Combined  DT/OT  Initiatives:  NAVSEA 

Supporting  CFFC  SEA  TRIAL 


Naval  Systems  Engineering  Directorate 


Virtual  SYSCOM  Sea  Trial 
Collaboration  Team 

> NAVSEA,  NAVAIR,  and  SPAWAR,  with 
support  from  NWDC 

>  Share  knowledge  of  the  SYSCOMs’ 
Experimentation  processes,  tools  and 
venues 

3rd  Fleet  conducted  a  Sea  Trial 
Symposium  for  Sea  Shield 
Experimentation 

^Identified  Fleet  Warfighting  gaps 

>  Aligned  and  prioritized  experimentation 
documented  in  SUMS 


NAVSEA  develops  initiatives  to  rapidly 
solve  gaps  and  cost 

>Provide  Engineering  review  of  initiatives  and 
gaps 

> Provide  operation  &  system  architecture  views 

PEO  PM’s  provide  DT  &  OT  event  activity 
to  NWDC 

NWDC  vet  experiments  through 
SYSCOM’s  prior  to  Fleet  Collaborative 
Team  action* 

>Support  Sea  Trial  Information  Management 
System  (SUMS) 


>Use  experimentation  to  fix  the  short  term 

gaps  Sea  Strike  -  2nd  fleet  -  NAVAIR 

Sea  Shield  -  3rd  Fleet  -  NAVSEA 

> Provide  representation  to  the  STESG  Forcenet .  NETWARCOM  -  SPAWAR 

Sea  Base  -  2nd  Fleet  -  NAVAIR 


Naval  Systems  Engineering  Directorate 


Distributed  Engineering  Plant  Overview 

(NSWCDD) 


AEGIS  BiS.3, 7.1  MAP, 
PAC3  BMD  Lab  ConriKt. 


Air  Force  f  lodu- 


OpipFalions 
Cwrtor  CTOCJ  & 
Network 
Operations 
Center  (NOC) 
San  Git go4  CA 


CV/CVN  Class 
LHA/LKD  Class 
ODfFFG  Class 

VSEAf  Dam  Neck 


CVKVN  Class 
LHA/L  HD  Class 
S SOS  MR  2 

IG5TF  -  San.  Diego 


C4ISR 

SSC-Ssin  Dleo? 


US  Army 


The  Distributed  Engineering  Plant  (DEP) 

-  Critical  element  of  the  Navy’s  response 
to  Battle  Force  Interoperability 

-  Formed  by  Federating  Combat  System 
Sites  Dispersed  Around  the  United  States 

-  HITL  CS  suite  to  evaluate  how  systems 
interact  with  CEC  and  TADIL 
environments 

-  All  ship  baselines  require  DEP 
evaluations  prior  to  deployment 

-  A  High-Fidelity,  Shore-Based  distributed 
Force  Test  bed 

-  Demonstrated  utility  for  Industry 
participation 

-  Demonstrated  utility  for  industry 
participation 

-  Established  foundation  for  and 
compatible  with  JDEP 

Primary  mission  to  provide  Shore-based 
Force-level  testing  of  deploying 


E-2C  Gil  a  HE  3000 

NAWCAD -PAX  River 


CSGs/ESGs. 

-  Force  Interoperability  Assessment 


The  mission  has  evolved  to  support  the 
entire  acquisition  cycle. 

-  Force-Level  Performance 

-  Prototype  Evaluation 

-  Developmental  Systems 

-  Force  Problem  Resolution 


DEP  Goal  is  to  Enable  Navy  Acquisition 
Decisions  Based  on  Sound  Force  System 

Engineering 
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Distributed  Engineering  Vision 


Naval  Systems  Engineering  Directorate 


Navy  Joint  and  Coalition 
Interoperability  Challenge 


ft 

O 

o 


£ 


Navy  DEP  -HWILfor 
Deployers  (13  Nodes) 


1998 


i 


em 

* 


Coalition  Distributed 
Testing  (2007) 

8  MTMD  Nations 


•UK 

•Germany 

•Netherlands 

•Australia 


US 

Spain 

Italy 


C/DIT  Testing 

Phase  2 


Phase  3&4 


Phase  I 


(2005) 

Fix  developing 


(2006/7)  -  IABM 
Testing  in  Joint 
Engineering 
Environment 


Tjk  awacs 

•PATRIOT 

•TAOM 

•SSDS 

•AEGIS  (2) 

•ACDS 

•E-2C 

•F-18 


Joint  Test  Env.: 
(2001)  Joint  I/O 

Capabilities  & 
Limitations  (4 
Joint  Nodes) 


(2003) 

Validate 

SIAP 

algorithms 
in  legacy 
•Patriot 
•AEGIS 
•E-2C 


Implementation  §IAP  Pathfinders  and  Connectivity: 
of  MIL  STD  System^; Interface  Description  (SV-1) 

ARC-210  (tentative) 


601 6C 
•Patriot 
•AEGIS 
•E-2C 


HOSTILE 
Patrio^Li 


N  (SSDS: 


Advanced^enti^^iviRAAM 


2001 


i 


2005 


t 


2006/7 


2010 


2015 


t 


BCS 


Joint,  Distributed  Engineering  finds  problems  early,  reduces  costs,  and 
improves  interoperability:  the  “Force  Multiplier” 
for  the  1,000-ship  Navy. 


Integral  Fire  07 

(AF-ICE,  USJFCOM  J89,  SIMAF,  JMETC,  JC2  JCAS,  JTEM,  NSWC  DN) 


Naval  Systems  Engineering  Directorate 


Description:  establish  a  Joint  persistent  test 
environment  to  support  Joint  Command  and  Control 
(JC2),  Joint  Close  Air  Support  (JCAS  )  Assessment  for 
USJFCOM;  and  Air  Force  Secretary  on  the  Warfighter 
Warfighter  Forwarder  (WWF)  integration 
Objectives: 

•  Conduct  a  technical  assessment  of  JCAS  in 
response  to  real  time  requests. 

•  Evaluate  Methods  and  Processes  C2  airspace  de- 
confliction  between  Army  NLOS  and  Net  enabled 
weapon  (NEW)  (Air  Force  and  Navy). 

•  Machine  to  Machine  re-tasking  of  strike  platforms 
and  NEW  from  the  Air  Operations  Center  (AOC). 


JTEM  (Non  Line  of  Sight  NEW)  Warplan 
Warfighter  Forwarder 


J  *  Airspace  De-Confliction 

•  NLOS  for  Joint  Network  Fires 

•  Link16  J11.X  messages  for  weapons  control  and 
handoff 

•  Tactical  Edge  Control  Utilizing  Link16  J9.0,  J11.X 
J12.0  messages 


is,  ~~  " 

V; 

A 

and 

JSIC  JC2  JCAS:  Assess  data  exchange  between 
AF/Army,  Marine,  and  SOF  Terminal  Air  Controller 


LVC  JME  Infrastructure 


Existing  Connections 
used  by  JNTC  (not 
be  used  in  07-03) 


E1  =  TacLane  with  JCAS  Key 
E2  =  TacLane  with  AF-ICE  Key 
E3  =  TacLane  with  SDREN  Key  | 
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T&E  Facility  Examples 
Ocean  &  Coastal  Ranges  and  Facilities 


SFTF 

□  AUTEC 


Pacific  Northwest 
Range  Complex 


Narragansett  Bay 
Shallow  Water  Test 
Facility 


Potomac  River  Test 
Range 


PNW  RC 


Surface  Combat 
Systems  Center 


Southern  California 
Offshore  Range 


Combatant  Craft 
Division 


South  Florida  Test 
Facility 


NBSWTF 
CSC 
PRTR 


CCD 


SCORE 


PMRF  Q 


Pacific  Missile 
Range  Facility 


^  Fleet  Ranges  ^  T&E  Ranges  Q  T&E/w  MRTFB  Components 


Gulf  of  Mexico 
Coastal  Test  Range 


Atlantic  Undersea  Test 
and  Evaluation  Center 


Naval  Systems  Engineering  Directorate 


Initiative:  Undersea  Ranges  Consolidation 
&  Collaboration  (NUWCKPT,  NPT) 


Portable  Range  Programs  Reduced 
From  12  to  5 
Cost  Avoidance  ~$3M 


Inventoried  East  Coast  Range 
Cable  given  to  KPT’s  Nanoose 
Range  for  Repair 
Cost  Savings  $800K 


Sponsors:  NRO 
Partners:  PMA205,  SCORE,  PMRF, 
LMC/L3/Maripro 


Leveraging  Existing  Systems  Future  Multi-use  Cost  Savings  TBD 


Common  Equipment 
Reduce  Cost 


One  Contract  for  Upgrades  at  2  Ranges  x 
Cost  Avoidance  ~$9M 


Assess  Using  IMPASS  (Gun  Firing)  System 
for  Portable  Surface  Ship  Radiated  Noise 
Measurements 


Collaboration 

Pinger  Consolidation  Across 
Warfare  Centers 
Cost  Savings  =  $150K/yr 


Common  Design 


Eliminating  Overlap 


USW  T&E  and  Training 


Ranges  Master  Plan 


SSA standup  SOAR  BSURE 

FYOO  01  02  03  04  initial  Roadmap  08  Complete  12  13-16  17-21  22+ 


Naval  Systems  Engineering  Directorate 


Undersea 


Tracking  Range  Roadmap 


•  Comprehensive  roadmap  of  USW 
tracking  range  requirements  (T&E  & 
Training) 

•  Held  UTR  T&E  requirements  forum  with 
PEO’s,  COTF,  FFC 

•  Provides  investment  guidance  for  near, 
mid,  and  long  term 

•  Identifies  shortfalls  or  gaps  in  resources 
and  shared  approaches  to  USW  ranges 
technology 

•  Completed  Phase  1,  conducting  Phase  2 

•  Documents  efforts  required  for  the 
current  ranges  out  to  FY20. 

•  Proposes  potential  technologies  and 
range  architecture 


DISTRIBUTION  AUTHORIZED 
OTHER  REQUESTS  MUST  BE 
WARFARE  CENTER  DIVISION 


1.  The  Undersea  Warfare  Analysis  and  Assessment  Product  Area 
(USW  AA  PA) ,  in  conjunction  with  members  of  the  undersea  range 
community,  have  developed  the  attached  Undersea  Tracking  Range 
Roadmap  Spiral  1.  Enclosure  'll  focuses  on  requirements 
relating  to  US  Navy  Undersea  Tracking  Ranges  lUTR'si  in  support 
of  Fleet  tactical  training,  and  comprised  three  major 

cfc; actives;  (1)  document  the  existing  undersea  tracking 
baselines  for  Navy  UTR's;  (2)  describe  the  range  acquisition 
programs  due  to  complete  by  2011;  and  (3)  develop  a  strategic 
plan  and  roadmap  for  UTR' s  beyond  2011,  focusing  on  technology 
and  process,  leveraging  and  cost  avoidance. 

2.  A  utre  spiral  2  effort  is  currently  underway.  This  spiral 

effort  will  provide  comprehensive  documentation  of  Test  and 
Evaluation  requirements,  conduct  further  analysis  of  the  impact 
of  these  requirements  on  range  technology  development,  and 
develop  appropriate  coordinated  strategic  Investment  and 
maintenance  planning  for  US  Navy  UTR's.  This  analysis  will 
include  a  detailed  examination  of  all  future  Undersea  Warfare 
(USW)  programs  being  developed  by  the  various  Program  Executive 
Offices  and  being  executed  by  the  various  Program  Managers  sea, 
and  Program  Managers  Air.  The  effort  shall  include  periodic 
stakeholder  meetings  ar.d  industry  day  briefings,  enabling 
industry  and  stakeholder  input,  and  facilitates  stakeholder 
review  and  consensus  building. 


3.  The  UTRR  is  expected  to  be  a  living  document  to  be  updated 
periodically.  The  UTRR  will  be  a  top-level  document  comprising 
a  comprehensive  long  term  investment  and  maintenance  planning 
strategy  for  USW  tracking  ranges.  It  will  be  fully  integrated 
with  USW  test  and  USW  training  ar.d  will  be  used  to  provide 
investment  guidance.  Questions  concerning  the  UTRR  may  be 
dr recced  to  the  following  rr.der??e  range  customer  advocates;  for 
Sprral  !,  Robert  Reid  at  101  :31-5797  and  for  Spiral  2,  Trevor 


Initiative:  White  Sands  Missile  Range  (WSMR) 

Desert  Ship  Upgrade 


Naval  Systems  Engineering  Directorate 


5 


•  Transitioned  20-40  year  old  specialized  Fire  Control 
System  (FCS)  equipment  to  an  Aegis  Open  Architecture 
FCS  and  COTS  based  approach. 

•  Supports  current  and  future  Integrated  Fire  Control  (IFC) 
tests  to  demonstrate  Naval  and  Joint  Capabilities 

•  Reduces  overall  Desert  Ship  system  maintenance  cost 

•  OA  and  COTS  approach  reduces  future  upgrade  costs 
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Initiative:  White  Sands  Missile  Range  (WSMR) 

Desert  Ship  Upgrade 


Naval  Systems  Engineering  Directorate 


Potential  Future  IFC  Test  Configurations 


The  Cooperative  Engagement  Capability  (CEC)  Land  Mobile 
Unit  (LMU)  is  connected  to  Desert  Ship  Aegis  OA  FCS  via  the 
White  Sands  Missile  Range  (WSMR)  Technical  Support 
Network.  There  are  numerous  connection  points  through  the 
entire  range  where  the  CEC  LMU  may  be  connected. 


WSMR  provides 
various  overland 
test  environments 

•  Desert 

•  Mountain 


Salinas  Peak 
MK-74  Radar 


CEC  LMU 


Initiative:  AAW  SSD  T&E  Enterprise 


SEA  05 


Naval  Systems  Engineering  Directorate 


•  PEO  IWS  Initiative  to  apply  an  enterprise 
approach  to  program  T&E  strategies 

•  Integrates  SDTS  and  Lead  ship  at-sea  test 
events,  and  Pra  Testbed  across  combat 
system  variants 

-Applicable  to  LPD  17,  LHA  6,  DDG  1000, 
CVN  78,  LCS 

-SSDS,  RAM,  ESSM 

•  Eliminates  duplication  and  optimizes  testing; 
element  TEMP(s)  still  supported 


Using  an  Enterprise  Approach 

FY  06/07  FY  08/09  FY 10/11  FY 11/12  FY 12/13  FY 14/15  FY 15/16 


ESSM 


Self  Defense  Test  Ship 

ENTERPRISE  Strategy  Configuration 


LCS 

BOFORS 


SPS-48E 

(TBD) 


LCS  ELECTRO-OPTIC 
DIRECTORS 


LCS 


RADAR 1&2 


SPY-3 

MFR 


NSSMSMK29  MOD  4 
LAUNCHER 


NSSMS 
MK  57  MOD  12 


SACEP 
USG-2  CEP 


VLS 

MK  41  /  MK  57 
HYBRID 


TSCE  CORE 


LCS  COMBAT 
SYSTEM  MODULE  1&2 


SSDS 

MK  2  MOD 3B 


SPS-49A(V)1 


Current 

Configuration 

New 


Test  Ship 

Concept  of  Oiieratlons 


ij&siii 


zJvji  i  Jlrwhls  hikjbJ 


Potential  savings  of  $240  Million,  and  reduction  of  38  missiles 


WAF-SMMTT  Value  to  the  Warfighter 


71 


SUBSCOL  NL  and 
NSTCP  SMMTT 


Proven 
Connectivity 
HLA  Interface 
ISDN 
or  use 
SIPRNETJ1 


NUWC  Simulation  and  Analysis 
Center  for  Torpedoes  and  CMs 


•  Real  Torpedo  Hardware  in  the  loop  and  latest  weapon  software 

•  High  fidelity  threat  models,  CMs,  and  high  fidelity  tactically  significant 
environments 

•  An  excellent  test  bed  to  preview,  and  test  Combat  Control  and  APB  Weapon 
System  and  HSI  improvements  prior  to  costly  at-sea  firings 

•  Proven  HLA  connectivity 

•  Leverage  the  latest  exploitation  for  training  in  the  employment  of  the  latest 
torpedoes  in  operationally  relevant  scenarios.  Used  as  part  of  SCC  training 
since  Oct  2004.  Utilized  for  mission-specific  Pre-Deployment  Training 
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Initiative:  NAVSEA  T&E  Human  Capital 

Strategy 


Naval  Systems  Engineering  Directorate 


WORK 

ASSIGM4HTT 


W-iat  wort  reeds  to  be  dine? 
•Technical  Gspefcility 
"Technical  Warrant 
•New  KroAfedge  Areas 


Workload 


WORKLOAD 

ACTIONS 


H  ean 

■Workload  Shading 

•CXjtsDurdng 

•Divestment 


WORKFORCE 

ASSESSMENT 

What  wrrfc  ran  hr.  donr? 
•  Icdt  liuil  U*xi)ilily 
«Kj  wwk.tjyu  /\i  Lva 
<0Tcctcncicc 

•rVTvyTrflnhlrr 

fuma-*w  OjILu  « 


Workforce 


Required  Effect 


RIGHT  WORK 
RIG*  IT  PLACE 
RIGHT  PEOPLE 
RICH  I  TLVf- 
RKtf  II  CI  A  INPI 


WORKFORCE 

SHAPING 


-Redeployment 

•Hiring! 

-Training  &  Dcrveiopment 
•Performance  Management 
•Motivation  a  Rewards 
•Diversity 


Metrics 


ASSESSING  WORKFORCE  HEALTH 


Measure  TC  Health  by 
Knowledge  Area  (KA) 

(e.g.  Test  &  Evaluation) 


-  Total  Knowledge  Stewardship  Need 

-  Total  Knowledge  Stewardship  Have  (Base  +  Pipeline) 

-  Knowledge  Base  Have 


FY07 


Gap  Assessment 


ASSESSING  WORKLOAD  DEMAND 


WORKFORCE  SHAPING  ACTIONS 


Assign  Workload  by  Technical  Capability  (TC) 


Yields 


WC  Enterprise 

Total  Direct  Workload  Demand 


w  18000 
re  17500 
>2  17000 
£  16500 

o  16000 

Z  15500 
g>  15000 
5  14500 
14000 
o  13500 
13000 

FY07  FY08  FY09 


Workload  Trends 


Hire  Or 
Redeploy 
based  on 
TC&KA 
Health 


Human  Capital 


FY07  Workforce  Hiring 
by  Site  TC 

FY06  Health 

Assessment 

Planned 

Current  YTD 

S&T 

5 

1 

TCI 

30 

12 

TC2 

20 

0 

TCX 

5 

0 

TCY 

4 

0 

Business  Operations 

5 

1 

Total 

69 

14 

|Directions:  Direct  S&T  Work  is  counted  in  appropriate  TC 

Entry  J  ourney  Expert  Senior 

Executive 


Develop  by 
5VM 

Career  Path 

(e.g.  Test  &  Evaluation) 
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Naval  Systems  Engineering  Directorate 


Counter  IED  Device  Testing 

,(NSWC  Corona,  Dahlgren,  NAVEODTECHDIV,  PMS408) 


Challenges 


Tasking 

Assessment  of  effectiveness  of  CREW 
systems  for  dismounted  troops,  wheeled 
vehicles,  and  riverine  patrol  boats. 

•  Quick  Reaction  Mounted  &  Dismounted 
CREW  -  Counter  Remote  Control  Improvised 
Explosive  Device  Electronic  Warfare 


Accomplishments 


8Compressed  acquisition  process  requires 
demanding  OPTEMPO. 

•  Transform  process  for  assessment  of  NAVY 
weapons  systems  for  assessment  of 
systems  used  in  ground  combat  environment 
to  counter  asymmetric  threat  -  lED’s. 


•  On-site  at  Yuma  Proving  Grounds  Az,  for 
data  collection,  analysis  &  operator  support 

•  Author  Effectiveness  reports 

•  Testified  as  SME  at  source  selection 
board  Feb  07. 
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Naval  Systems  Engineering  Directorate 


M&S  Initiative:  Envisioned  Roadmap 
Towards  M&S  of  the  Full  Ship 

Shock  Trial 


Business 

Strategies 


M&S 

Technologies 


T&E 

Engineering 

Basic  Science 
of  Material 


■Uncoupled  simulators  of 
shock  load. 

■Global  energy  equations 
(include  fragment  energy). 


•Certify  LSDYNA, 
ABAQUS,  NASTRAN, 
DYSMAS..  in  accordance 
with  specified  capabilities 
and  limitations. 


•  Collaborative  Investr 

most  critical  enablers., 


Integrated 
M&S  of^SSl 


•  End  to  end  coupl^cfM&S  Mth  imbedded 
beam  analogy,  joirtt .analogy;  cavitation  load 
analogy,  jragment  ^energy  balance  models. 


Structure  Dynamics  Under  Shock  /Blast  load. 

•Beam-column,  Pipe-structure  connection,  Bolted 
joints. 

•Shock  transmission  across  joints.  Fracture  &  fragment 
Fluid-Structure  Interface  Modeling 

•CFD  Modeling  of  blast  intrusion. 

•CFD  cavitation  interaction  with  structure. 

•Adaptive  FEB  for  fluid  structures.  *  IntegratecHINBEX  whipping  resistano 
Numerical  Models  ___assfiS8mSnti model. 


Account/or  effects  of  as-builts 
for  accuracy 

use  or  imaging  0oth  EM  and 
acoustic)  for  direct,  near-automatic 
tscretization  &/meshing. 


•Non-reflecting  B.C.’s  ,  T7^  ,  •Alternate  T&E  to  support  critical  gatfs. 
•PeaK  pressure,  total  impulse  sensitivity  calibration. 

FSST«Fast  transient  response  instrumentation. 

•Pseudo  velocity  shock  spectrum. 

High  speed  actuator  for  shock  testing,  Multi-axis  transducers. 
Visco-elastic,  non-linear,  non-isotropic,  strain  rate-stress  constitutive  models. 
Composite  failure  models,  fracture  mechanics. 


2006 


l 

2011 


2016 
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Future  of  NAVSEA  T&E 


Naval  Systems  Engineering  Directorate 


Mature  Test  Evaluation  &  Analysis  Competency 

•  Support  Navy  T&E  BOD  integrated  investment  strategies 

•  Create  end-to-end  transparency  in  demand  signals,  improve 
execution  capacity  and  efficiency,  reduce  cost  of  execution  and 
ultimate  deliverables,  and  improve  customer  satisfaction 

Enable  System  Engineering  To  Support  Acquisition  via 
Integrated  Strategic  Planning 

•  Emphasize  enterprise  integrated  solutions  versus  platform  or 
singular  domain  focus  across  PEO  T&E  Directors,  HQ  &  Warfare 
Centers  T&E  Executive  activities 

•  Enable  Navy  T&E  in  a  Joint  Environment 

•  Enable  Affordable  T&E  of  Open  Architecture  Systems 
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Perspectives  on  Acquisition, 
Test,  and  Early  Fielding  of  UAV 

Systems 

Curt  Cook 
DOT&E 


Perspectives 


•  ACTD  legacy:  Departure  from  the  intent  of  ACTD  process  in  most 
UAS  programs  set  stage  for  numerous  negative  consequences 
affecting  the  systems’  performance  and  pace  of  maturity. 

•  Dual-program  problem:  Development  of  the  intended  program 
continued  concurrent  with  early  fielding  of  immature  systems.  This 
created  a  dual-program  problem  (a  deployed  system  and  an  intended 
system,  both  under  development). 

•  Limited  meaningful  testing:  As  acquisition  programs  of  the 
intended  system  took  shape,  service  acquisition  leadership  and 
program  management  demonstrated  an  aversion  to  end-to-end, 
mission  level  OT&E  (excepting  few  programs). 

•  Performance:  The  single  most  important  performance  characteristic 
for  unmanned  systems  should  have  been  high  reliability — this  has  not 
been  the  case. 


What  should  be  done  now? 


Fielded  systems:  Invest  in  reliability  growth  and  improvements  to 
suitability. 

Any  other  performance  improvements  must  positively  contribute  to 
improved  reliability  and  availability  of  the  overall  system. 

Defend  costs  for  spares,  maintainability  improvements,  and 
deployability  of  systems. 

Improve  initial  CONOPS:  Document  and  spread  the  word  on  what 
worked,  what  didn’t  work,  and  what  needs  to  be  fixed  in  the 
integration  of  UASs  with  ground  and  air  combat  units. 

Systems  under  development:  Accomplish  the  fundamentals 
necessary  for  a  successful  program: 

Ensure  sufficient  requirements  traceability 
Re-evaluate  requirements  for  reliability  and  maintainability 
Complete  and  validate  concept  of  operations  and  support 

Plan,  resource  and  complete  adequate  developmental  and 
operational  testing 


Probability  of  Completing  a  24  Hour 
Mission  Without  a  Failure 


Typical  UAV 
systems 


ETOS 


Effect  of  Air  Aborts  on 
Effective  Time  On  Station 


Assumes: 

•100  percent  availability 
•Relief  on  Station  (unlimited 
aircraft  simultaneously  airborne) 
•Instantaneous  launch  on 
request  (no  start  up  time) 

•No  ground  aborts 
•Exponential  air  abort 
distribution 

Shows  upper  bound  on  ETOS 
imposed  by  air  aborts 


FY01 

FY02 

FY03 

FY04 

FY05 

FY06 

FY07 

FY08 

FY09 

FY10 

FY1 1 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

1Ql2Ql3Ql4Q 

A 

MS-II 


Feb  01  iot&e- 

63  RQ-4A  air  vehicles,  14  ground  segments 


A 

BlklO  LRIP 


12  w/  basic  EO/IR  +  basic  SAR  (Block  5) - 

26  w/  imp  EO/IR  +  RTIP  SAR  (Block  10) 

25  w/  SIGINT  (Block  10) 

$85. 6M  PAUC,  $70.8M  APUC  (excludes  ACTD  $574M  and  7  a/c) 
LRIP  approved:  6  Blk  5  air  vehicles,  2  ground  segments 


1st  restructure 


CD 


A 

IPR 


Mar  02 


9  RQ-4A,  42  RQ-4B,  10  ground  segments 
9  RQ-4A  w/  various  configurations:  basic  or  improved  EO/IR: 

basic  or  improved  SAR 
4  RQ-4B  w/  various  configurations 
38  RQ-4B  w/imp  EO/IR  +  RTIP  SAR  +  SIGINT 
S134.2M  PAUC,  $86. 0M  APUC 
LRIP  approved:  17  air  vehicles,  4  ground  segments 


2nd  restructure 

Dec  02 


A 

IPR 


7  RQ-4A,  44  RQ-4B,  10  ground  segments 
7  RQ-4A  w/  various  configurations 
7  RQ-4B  w/  various  configurations 

12  RQ-4B  w /  MP-RTIP  only  (RTIP  a/c  start  in  09  instead  of  07) 
25  RQ-4B  w/  imp  EO/IR  +  imp  SAR  +  SIGINT 
$1 14.0M  PAUC,  $64. 2M  APUC 

LRIP  approved:  19  air  vehicles,  4  ground  segments 


3rd  restructure 

Marj 


□  A 


IOT&E-2  BlklOFRP 

$120M 


SIGINT  FOT&E  and  IOT&E-2  on  MP-RTIP 
(TEMP  w/  dates  approved  after  2nd  restructure) 


Global  Hawk 
Evolution 


RQ-4A  IMINT 


/\  RQ-4B  SIGINT  RQ-4B  MP-RTIP 

DR  FOT&E  IOT&E-2 


RQ-4A  IMINT 


IPR  ha 


$12M 


ProcuremenT 
(24  mo.  to  delivery) 


Same  production  plans  as  2nd  restructure, with  cost  increase,  later  delivery 
Reduce  capability  of  RQ-4A  (no  <12hr  mission  planning) 

Field  2  production  aircraft/sensors  to  replace  deployed  ACTD  assets  ($20M  unfunded) 
Delay  tests  b/c  of  late  development  (OA  ~1  year,  IOT&E  2+  years) 

Est.  $125M  PAUC,  $64M  APUC,  (b)ased  on  11%  over  Dec02  APB  threshold,) 

Delayed  IOT&E/BLRIP  delays  FRP  decision 

LRIP  approved:  40  air  vehicles.  12  ground  segments 


ARTipn  a 

IOT&E  FRP  IOT&E  RTIP 
FRP 


2  AV 
1  GS 


6  AV 
(1  USN) 
2  GS 


4  AV 
1  GS 


4  AV 
1  GS 


5  AV 
1  GS 


5  AV 
3  GS 


5  AV 
1  GS 


5  AV 
2  GS 


5  AV 
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$80M 


$60M 


$40M 


$20M 


10  AV 
TC 


Major  New  Initiatives  Affecting  T&E 


•  McQueary/Young  Policy  Provisions 


•  McQueary  Goals  and  Priorities 


•  New  Approaches  for  System  Development,  Testing  and  Fielding 


McQueary/Young  Policy  Revisions 


•  Fundamental  Purpose  of  T&E  :  Provide  Knowledge  to  Assist  in 
Managing  Risks  in  System/Capability  Development,  Production, 
Operations  &  Sustainment 

-  Measure  Progress  in  Both  System  &  Capability  Development 

-  Provide  Knowledge  of  System  Capabilities  &  Limitations  to  the 
Acquisition  and  User  Communities 

-  Bring  to  Bear  Expertise  at  the  Outset  of  a  System’s  Life  Cycle 

-  Identify  Technical,  Operational  and  System  Deficiencies  Early 


McQueary/Young  Policy  Revisions 


DT&E  and  OT&E  Integrated  &  Seamless  throughout  Life  Cycle 

Evaluations  Include  Comparison  with  Current  Mission  Capabilities  to 
Determine  Measurable  Improvements 

Assess  Improvements  to  Mission  Capability  and  Supportability  Based  on 
User  Needs  --  Report  in  Terms  of  Operational  Significance 

Evaluations  Include  All  Available  &  Relevant  Data 

Conduct  T&E  in  a  Continuum  of  Live,  Virtual  &  Constructive  Environments 


McQueary  Goals  and  Priorities 

Improve  Suitability 

Enhance  Operational  Realism  in  Early  Tests,  including  DT&E 
Provide  Timely  Performance  Information  to  the  Warfighters 
Conduct  Mission-Based  (End-to-End)  OT&E 

Improve  Operational  Test  Resources 

New  Approaches  for  System  Development,  Testing  and  Fielding 

Competitive  Prototyping 
Capabilities-Based  Acquisition 

Rapid  Acquisition  of  Systems  driven  by  Urgent  GWOT  Requirements 


New  Emphasis  on  Testing  in  a  Joint  Environment 


Challenges  for 
Test  and  Evaluation  (T&E) 
in  the  Defense  System  of  Systems 

Environment 

Dr.  J  udith  Dahmann 
The  Ml  TRE  Corporation 

Darlene  Mosser-  Kerner 
OUSD  AT&L  SSE/  DTE 


System  of  Systems: 

A  set  or  arrangement  of  systems  that  results  when  independent  and  useful  systems  are  integrated  into  a 
larger  system  that  delivers  unique  capabilities.  DoD  Defense  Acquisition  Guide 


Why  Systems  of  Systems  (SoS)? 


•  SE  is  a  critical  enabler  for  systems  acquisition 

♦  Today's  acquisition  process  focuses  on  the  development  of  systems 

•  Most  military  systems  today  are  part  of  an  SoS 
whether  or  not  explicitly  recognized 

♦  Most  systems  are  created  &  evolve  without  explicit  SE  at  the  SoS 
level 

♦  Traditionally  the  combination  of  systems  to  meet  operations  has 
been  in  context  of  operations 

•  Given  increased  networking  and  ranges  of  both 
sensors  and  weapons,  it  is  becoming  important  to 
recognize  SoS  in  development  and  systems 
engineering 

♦  SoS  dependencies  have  an  impact  on  development  and  engineering 
of  systems 

♦  DoD  is  recognizing  SoS  from  both  a  management  and  engineering 
perspective 


March  3,  2008 


2 


Obj  ecti  ve 


•  Ensure  the  DoD  SE  community  is  equipped 
to  support  war  fighting  capabilities  at  the 
system,  system  of  systems,  and  enterprise 
levels  by 

•  Understanding  the  nature  of  the  current  and 
emerging  development  environments  and  the 
challenges  they  pose  for  SE 

•  I  dentifying  best  practices  in  conduct  of  SE  in 
supporting  development  and  acquisition 

•  Providing  enabling  policy,  guidance,  education 
and  training 
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What  Are  We  Doing  with  SoS  and  SE? 


•  Completed  SoS  SE  Guide  v.1.0  in  December  2007 

•  Executed  six  month  'pilot  phase' 

♦  SoS  practitioners,  research  teams  and  industry 

♦  Structured  walkthrough  of  the  draft  guide  contents  to  capture 
their  experience 

•  Pilot  results 

♦  I  dentified  key  SoS  SE  elements  and  principles 

♦  I  dentified  SoS  SE  issues  which  require  further  attention 

•  Socializing  results  (i  ncose,  i  eee,  ndi a,  others) 

•  Next  steps 

♦  Update  DoD  SE  Guides  (SEP,  DAG)  for  SoS  considerations 

♦  Plan  for  DAU  Continuous  Learning  Module  in  FY08 

♦  I  mplement  FY08  activities  to  address  identified  issues 

•  Testing 


A  mechanism  to  share  emerging  insights  on  SoS  and  implications  for  SE 
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Pilot  Participants 


Objective  of  the  pilots 
was  to  gain  a 
'boots  on  the  ground' 
perspective 


Researchers/  FFRDCs 

I NCOSE:  International  Council  on  SE 
MIT:  Massachusetts  I  nstitute  of  Technology 
MITRE:  MITRE  Corporation 
Purdue:  School  of  Engineering 
SEI :  Software  Engineering  Institute 
Stevens:  Institute  of  Technology 
USC:  University  of  Southern  California 
UCSD:  University  of  California  San  Diego 

NDIA:  National  Defense  I  ndustry  Assoc. 
Australia:  Defence  Materiel  Organisation 


SE  Practitioners 

ABCS:  Army  Battle  Command  System 

AOC:  Air  Operations  Center 

BMDS:  Ballistic  Missile  Defense  System 

CAC2S:  Common  Aviation  Command  &  Control  System 

DCGS-AF:  Distributed  Common  Ground  Station 

DoDII  S:  DoD  Intelligence  Information  System 

FCS:  Future  Combat  Systems 

Ml  LSATCOM:  Military  Satellite  Communications 

Nl  FC-CA:  Naval  Integrated  Fire  Control  -  Counter  Air 

SR:  Space  Radar 

NSA:  National  Security  Agency 

NSWC:  Naval  Surface  Warfare  Center  Dahlgren 

PEO  GCS:  Ground  Combat  Systems 

SIAP:  Single  I  nteg rated  Air  Picture 

SMC:  Space  and  Missile  Systems  Center 

TMI  P:  Theater  Medical  Information  Systems  -  Joint 

USGC:  US  Coast  Guard  C2  Convergence 
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Emerging  Insights  from  SoS  Pilots 

SoS:  Is  It  New? 


•  Most  military  systems  today  are  part  of  an  SoS  whether  or  not 
explicitly  recognized 

•  Most  systems  are  created  and  evolve  without  explicit  SE  at  the  SoS 


level 


•  A  formal  SoS  comes  into  existence  when  something  occurs  to 
trigger  recognition  of  SoS 

•  An  organization  is  identified  as  'responsible  for1  the  SoS  'area' 
along  with  definition  of  the  objective  of  the  SoS 

•  Typically  does  not  include  changes  in  ownership  of  the  systems  in  the 


SoS 


•  The  SoS  is  then  structured 

<  Membership  is  defined  starting  with  identification  of  systems  in  the 


SoS 


•  Processes  and  organizations  are  established  for  the  SoS,  including  SE 
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What  Does  SoS  Look  Like  in 
the  DoD  Today? 


•  Typically  an  overlay  or  ensemble  of  individual  systems 
brought  together  to  satisfy  user  capability  needs 

•  Not  new  acquisitions  per  se 

♦  Cases  like  FCS  are  extremely  rare  and,  in  practice,  still  must 
integrate  with  legacy  systems 

•  SoS  'manager'  does  not  control  the  requirements  or 
funding  for  the  individual  systems 

♦  May  be  in  a  role  of  influencing  rather  than  directing,  impacts  SE 
approach 

•  Focus  of  SoS  is  on  evolution  of  capability  over  time 

•  A  functioning  SoS  takes  start-up  time  but,  in  steady 
state,  seems  well-suited  to  routine  incremental  updates 


March  3,  2008 
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Only  by  exception  do  we  manage  and  engineer  at  SoS  level 
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Relationship  Among  Core  Elements  /  /ns/9/}. 
ofSoSSE  _ LjSg 


Translating  SoS  capability 
objectives  into  high  level 
requirements  over  time 

Understanding  the  systems  in 
the  SoS  and  their  relationships 

Assessing  extent  to  which  the 
SoS  meets  capability  objectives 
over  time 

Developing,  evolving  and 
maintaining  a  design  for  the  SoS 

Anticipating  and  assessing 
impacts  of  potential  changes  on 
SoS  performance 

Evaluating  new  and  evolving 
requirements  on  SoS  and  options 
for  addressing  these 

Orchestrating  upgrades  to  SoS 


SoS 

upgrade 

process 


Persistent 
SoS  overlay 
framework 


External 

influences 


SoS  SE  creates  and  continually 
applies  approaches  to 
accomplish  these  elements 
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Where  Does  T&E  Fit  into  SoS  SE? 


r  ^ 

System  of  Systems 


\  / 


( 

Component 

Systems 

\ 

V 

•  Focus  of  SoS  engineering 
on  providing  end-to-end 
performance  that  meets 
capability  needs 

•  Within  the  context  of  the 
performance  requirements 
and  capability  limitations  of 
the  constituent  systems 


•  Focus  in  engineering  in  a 
single  system  environment 
on  optimizing  to  meet 
specific  performance 
objectives 
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Where  Does  T&E  Fit  into  SoS  SE? 
SoS 


•  SoS  metrics  and  methods  assess  capability  performance 
as  differentiated  from  capability  development 

•  SoS  metrics  need  to  focus  on  SoS  performance  instead  of 
program  execution  metrics,  and  on  the  intended 
integrated  behavior  and  performance  of  the  SoS  in  actual 
operations 
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Where  Does  T&E  Fit  into  SoS  SE? 
Systems 


•  Changes  in  systems  are  made  to  meet  SoS  objectives 

•  Typically  done  as  part  of  the  system  development  and 
test  process 

•  I  nclude  system  testing  as  well  integration  and  test 
across  sets  of  system 


Component  Systems 


^  - 

r  . 

Assessing 

1  ntegrating  and 

V 

Defining 

performance 

performance 

>.  testing  sets  of 

of  component 

of  component 

component 

i 

systems 

systems 

systems 

/ 

Orchestrating 
upgrades 
to  SoS 
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Where  Does  T&E  Fit  into  SoS  SE? 
SoS  and  Systems 


♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

Component  Systems 

- 

/  . 

-0 

Defining 

Assessing 

1  ntegrating  and 

performance 

performance  > 

testing  sets  of 

of  component 

of  component 

component 

systems 

systems 

systems 

7 

Orchestrating 
upgrades 
to  SoS 
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SoS  T&E  Assumptions 


•  Operational  I  ndependence  of  Systems 

♦  Each  system  has  a  useful  purpose  outside  of  the  SoS 

•  Managerial  I  ndependence  of  Systems 

♦  Each  system  is  acquired  and  sustained,  independent  of 
the  other  systems 

•  Evolutionary  Development  of  Systems 

♦  Each  system  is  developed  in  operationally  useful 
increments,  uncoordinated  with  the  other  systems 

•  SoS  provides  synergistic  capabilities 

♦  Unique  capabilities  not  available  with  a  single  system 

♦  Synergy  works  both  ways  -  positive  and  negative 
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SoS  T&E  Thoughts 


•  Traditional  requirements  process  leads  to  a  "point 
design" 

•  System  designs  are  optimized  and  tested  for  individual  system 
requirements 

•  SoS  applications  are  not  necessarily  aligned  with  initial 
system  purpose 

•  SoS  capability  relies  on  effective  interaction  between 
systems 

•  I  ncreases  emphasis  on  testing  the  collective  behavior  and 
sustainment  of  that  behavior 

•  Focus  on  system  capabilities  and  limitations  would 
provide  better  knowledge  base  for  SoS  application(s) 

•  Helps  identify  useful  SoS  capabilities 

•  Helps  identify  harmful  SoS  interactions 
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SoS  T&E  Challenges 


•  Clear  I  dentification  of  SoS  capability  requirements 

•  How  much  SoS  testing  is  needed? 

•  Piggy  back  on  system  test  (non-interference) 

•  Simulations 

•  Regression  of  SoS  capabilities  and  limitations 

•  SoS  Visibility  into  individual  system  design  and 
capabilities 

•  "White  Box"  vs  "Black  Box"  system  visibility 

•  Resolution  of  SoS  capability  or  limitation  issues 

•  Which  system(s)  to  change?  Trade  space 

•  Requires  establishment  of  a  value  construct  between 
individual  systems 

•  Timing  may  be  driven  by  opportunities 
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SoS  T&E  Challenges  (cont.) 


•  I  dentification  of  SoS  unique  test  risks 

♦  How  do  you  identify  synergistic  weaknesses? 

•  I  nf restructure  capability  to  test  SoS 

♦  May  require  networked  test  ranges,  labs,  models,  simulations 
(technical  issues) 

♦  May  require  coordinated  asset  times  (management  issues) 

♦  I  nstrumentation  capability  including  long  term  field  data 

♦  Data  management 

♦  Sharing  of  data  -  including  classified  and  proprietary  issues 

•  Evaluation  methodology  for  SoS 

♦  Can  we  scale  analysis  and  system  evaluation  methods  to  SoS 
size? 
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Next  Steps  for  SoS  T&E 


Objective 


•  Ensure  the  DoD  SE  community  is  equipped 
to  support  war  fighting  capabilities  at  the 
system,  system  of  systems,  and  enterprise 
levels  by 

♦  Understanding  the  nature  of  the  current  and 
emerging  development  environments  and  the 
challenges  they  pose  for  SE 

♦  I  dentifying  best  practices  in  conduct  of  SE  in 
supporting  development  and  acquisition 

♦  Providing  enabling  policy,  guidance,  education 
and  training 


Develop  a  robust 
understanding  of  the 
challenges  and  their 
sources 

Assess  current  initiatives 
and  approaches  and 

I  dentify  emerging 
principles  ancf 
opportunities  for  new 
initiatives 

♦  Remove  barriers  as 
necessary 
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Contact  I  nfo 


Dr.  J  udith  Dahmann 


Darlene  Mosser-  Kemer 


http://www.acq.osd.mil/sse 


Contact  us  to  provide  feedback  and  share  your  experience 
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Core  Elements  of  SoS  SE 


•  Translating  SoS  capability  objectives  into  high  level 
requirements  over  time 

•  Understanding  the  systems  in  the  SoS  and  their 
relationships 

•  Assessing  extent  to  which  the  SoS  meets  capability 
objectives  over  time 

•  Developing,  evolving  and  maintaining  a  design  for  the 
SoS 

•  Anticipating  and  assessing  impacts  of  potential 
changes  on  SoS  performance 

•  Evaluating  new  and  evolving  requirements  on  SoS  and 
options  for  addressing  these 

•  Orchestrating  upgrades  to  SoS 


The  SoS  SE  is  responsible  for  creation  and  continual  application 
of  approaches  to  accomplish  these  elements 
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DoD  SE  Community 


•  Ensure  the  DoD  SE  community  is  equipped  to  support  war 
fighting  capabilities  at  the  system,  system  of  systems,  and 
enterprise  levels  by 

♦  Understanding  the  nature  of  the  current  and  emerging 
development  environments  and  the  challenges  they  pose  for  SE 

♦  Identifying  best  practices  in  conduct  of  SE  in  supporting 
development  and  acquisition 

♦  Providing  enabling  policy,  guidance,  education  and  training 
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System  of  Systems  - 
The  Management  Challenge 


SoS: 

Within 

Single 

Organization 


Joint  SoS: 

Interdependencies 

Across 

Multiple 

Organizations 


Political  and  Cost  Considerations  I  mpact  on  Technical  I  ssues 
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SoS  Characteristics  Which  I  mpact 
Testing 


•  Difficulty  in  creating  measurable  SoS  objectives 

•  Continued  independent  ownership  of  requirements 
and  development  of  the  component  systems 

•  Variable  component  system  contexts 

•  Asynchronous  nature  of  the  development 
processes  across  the  SoS 

•  Size  and  variability  in  SoS  membership  and 
application  domains 

•  Scale  of  the  SoS 

•  Emergent  behaviors 
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OTA  Roundtable: 
Applying 
the 

T&E  Requirements  Process 

to 

Unmanned  /  Autonomous  Vehicles 


Stephen  C.  Daly 

Deputy  Director,  Operational  Test  &  Evaluation  for  Land  &  Expeditionary  Warfare 

Office  of  the  Secretary  of  Defense 

Pentagon  Room  1D548 

(703)  697-3891 

stephen.daly@osd.mil 


26  February  2008  -  Slide  1 


Deputy  Director,  Land  &  Expeditionary  Warfare 


Requirements 


•  Warfighters  write  requirements  or  “must  have”  capabilities 
-  not  testers,  not  evaluators. 

•  Warfighters  define  the  concept  of  operations  -  not 
testers,  not  evaluators. 

•  Program  Managers,  testers,  evaluators  need  to 
understand  the  RATIONALE  behind  the  warfighter’s 
requirements  and  the  warfighter’s  CONOPS. 

■  PM:  where  to  trade  cost,  schedule,  and  performance. 

■  Tester:  structure  tests  to  demonstrate  system  &  unit  performance. 

■  Evaluator: 

•  Assess  unit  mission  accomplishment. 

•  Assess  system  performance. 

•  Answer  the  “so-what”  question. 
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Comprehensive  Testing 


•  Works  in  a  well-known  process  to  demonstrate  system 
performance  in  a  “model-test-fix-test”  methodology. 

■  Developmental  Test  &  Evaluation. 

■  Live  Fire  Test  &  Evaluation. 

■  Operational  Test  &  Evaluation. 

■  Evaluation  of  the  significance  of  system  performance  &  unit  missior 
accomplishment  performance. 

•  Comprehensive  T&E  characterized  by  some  as: 

■  “Too  slow.” 

■  “Too  costly.” 

■  “You  don’t  understand  that  my  UAS/UAV  is  special.” 

■  “We’re  at  war  for  gosh  sakes.” 
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DOT&E  Point  of  View 


•  Many  past  systems  started  as  Advanced  Concept 
Technology  Demonstration  (ACTD). 

■  Execution  of  ACTD’s  often  skimps  on  the  demonstration. 

■  Often  fail  to  address  operational  suitability  issues. 

■  Often  has  a  cumbersome  transition  to  a  program-of-record. 


•  Common  theme  is  to  avoid  government  test  in  any  form. 


•  In  IOT&E: 

■  Some  UASs  have  been  assessed  as  operationally  effective. 

■  To  date,  none  have  been  assessed  as  operationally  suitable. 

■  Services  have  applied  lessons  learned  and  corrections  to 
improve  these  systems  since  IOT&E. 

■  Lessons  apply  to  other  Unmanned  Systems  in  development. 
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Shadow  IOTE  in  2002  at  Fort  Hood 


•  Operationally  effective  under  fair  weather  conditions  and  in  the 
absence  of  an  air  threat  for  cued  reconnaissance  and  surveillance 
missions  ...  not  operationally  effective  to  target  acquisition 
missions. 

■  57%  of  Recon-Surveillance  reports  were  timely  and  accurate. 

■  Demonstrated  target  location  error  in  excess  of  200  m  (80  m  req’d). 

•  Not  operationally  suitable. 

■  Not  reliable,  not  maintainable,  2  AV  crashes  and  1  significant  damage. 

■  Operational  Availability  meets  user  requirements  due  to  redundancy. 

■  Demonstrated  ability  to  meet  the  Commander’s  operational  tempo. 

•  Not  survivable. 

■  Susceptibility  to  detection  is  high. 

■  Seen  and  heard  within  effective  ranges  of  many  threats. 

■  Significant  electromagnetic  vulnerability. 
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Predator  IOTE  in  2000 


•  Predator  is  not  operationally  effective  or  suitable. 

■  Disparity  between  the  apparently  successful  fielded  system  and  the 
systems  that  did  not  perform  well  in  IOT  is  largely  attributable  to  the  fact 
that  the  system  is  tasked  and  operated  well  within  known  limitations 
such  as  effective  time-on-station,  weather  restrictions,  expected  threats, 
expected  accuracy,  and  dissemination  abilities. 

■  Capable  of  surveillance,  recon,  &  battle  damage  assessment  missions. 

■  Poor  Target  Location  Error  (TLE),  weather  restrictions,  and  ineffective 
communication  impact  strike  support,  CSAR,  area  search,  and 
continuous  coverage. 

■  Cannot  meet  requirements  outlined  in  the  ORD  &  KPPs. 

■  Lack  of  relief-on-station  procedures  and  poor  reliability  renders  the 
system  unable  to  meet  the  75%  Effective  Time-on-Station  (ETOS) 
requirement  at  range  of  400  nm. 

■  Serious  deficiencies  in  reliability,  maintainability,  and  human  factors. 
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Deputy  Director,  Land  &  Expeditionary  V\farfare 


Raven  IOTE  in  2006  at  Fort  Bliss 


•  Raven  SUAS  is  operationally  effective. 

■  Infantry  company  commander  benefits  from  enhanced  situational 
awareness  and  more  operational  planning  options. 

■  Used  effectively  in  lieu  of  manned  reconnaissance. 

■  Can  recognize  manned-sized  objects,  but  cannot  identify  armed  from 
unarmed  personnel  or  find  lEDs. 

■  AV  susceptible  to  acoustic  and  visual  detection. 

•  Raven  SUAS  is  not  operationally  suitable. 

■  SUAS-equipped  unit  cannot  sustain  itself  in  prolonged  combat. 

■  Consumed  parts  at  a  rate  in  excess  of  the  parts  allocation. 

■  Operators  were  able  to  quickly  repair  the  AV  if  parts  were  on-hand. 

■  AV  was  not  reliable,  demonstrated  5.6  hrs  MTBSA  (12  req’d). 
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Observations  &  Advice 


•  Focus  on  warfighter’s  mission  accomplishment,  CONOPS,  and 
total  life-cycle  cost. 

■  Understand  the  rationale  behind  the  capabilities  and  KPPs. 

■  Clearly  define  mission  success. 

•  Reliability . Reliability . Reliability. 

•  Take  advantage  of  inherent  redundancy,  balance  vs  workload. 

•  Some  tactical  UAS  fielded  without  operators  -  additional  duty. 

•  Target  location  error  and  resolution. 

•  Weapons  -  create  whole  new  set  of  additional  operational  issues. 

•  Work  to  reduce  UAS  susceptibility  in  early  design  efforts. 

•  System  proliferation  creates  airspace  management  issues  -  will 
face  similar  terrestrial  battlespace  management  issues. 

•  Extreme  Environments  -  hot,  cold,  wind,  weather. 

•  Countermeasures,  GPS  jamming,  friendly  Counter  IED  Systems  - 
complex  spectrum. 

•  Incorporate  Real-Time  Casualty  Assessment  System  (RTCA). 

•  Warfighter’s  operate  in  a  joint  environment  today. 
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Back-up  Slides 
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10  use  Section  139 


•  “There  is  a  Director  of  Operational  Test  and  Evaluation 
in  the  Department  of  Defense,  appointed  from  civilian 
life  by  the  President,  by  and  with  the  advice  and 
consent  of  the  Senate.” 


*  “Operational  test  and  evaluation  means  - 

■  the  field  test,  under  realistic  combat  conditions,  of  any  item  of 
(or  key  component  of)  weapons,  equipment,  or  munitions  for 
use  in  combat  by  typical  military  users;  and 

■  the  evaluation  of  the  results  of  such  test.” 


26  February  2008  -  Slide  10 


Deputy  Director,  Land  &  Expeditionary  V\farfare 


10  use  Section  139 


“The  Director  shall  -- 

•  prescribe,  by  authority  of  the  Secretary  of  Defense,  policies  and 
procedures  for  the  conduct  of  operational  test  and  evaluation  in  the 
Department  of  Defense; 

•  provide  guidance  to  and  consult  with  the  Secretary  of  Defense  and 
the  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
and  the  Secretaries  of  the  military  departments  in  general  and  with 
respect  to  specific  operational  test  and  evaluation 

•  coordinate  operational  testing  conducted  jointly  by  more  than  one 
military  department  or  defense  agency; 

•  monitor  and  review  all  operational  test  and  evaluation  in  the 
Department  of  Defense; 

•  review  and  make  recommendations  to  the  Secretary  of  Defense  on 
all  budgetary  and  financial  matters...; 

•  monitor  and  review  the  live  fire  testing  activities  of  the 
Department....” 
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Operational  Effectiveness 


The  overall  degree  of  mission  accomplishment  of  a 
system  when  used  by  representative  personnel  in  the 
environment  planned  or  expected  for  operational 
employment  of  the  system  considering: 

■  organization 

■  doctrine 

■  tactics 

■  survivability 

■  vulnerability 

■  threat 
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Operational  Suitability 


•  The  degree  to  which  a  system  can  be  satisfactorily 
placed  in  field  use,  with  consideration  given  to: 

■  availability 

■  compatibility 

■  transportability 

■  interoperability 

■  reliability 

■  wartime  usage  rates 

■  maintainability 

■  safety 

■  human  factors 

■  manpower  supportability 

■  logistics  supportability 

■  documentation 

■  training  requirements 
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Hon.  Charles  E.  McQueary 
Director,  Operational  Test  &  Evaluation 
703-697-3655 


David  W.  Duma 
Principal  Deputy  Director 
703-697-4813 


Dr.  Ernest  Seglie 

Science  Advisor 
Ernest.Seglie@osd.mil 
703-697-3655 

Missile  Defense  J 

r  COL  Bill  Bennett 

Senior  Military  Assistant 
Henry.Bennett@osd.mil 
v  703-697-3655  , 

j _  _ i _ 

_ i _  _ i _ 

Stephen  C.  Daly 
Deputy  Director 
Land  &  Expeditionary 
Warfare 
703-697-3891 
Steve.Daly@osd.mil 


Michael  D.  Crisp 
Deputy  Director 
Air  Warfare 
703-692-9929 
Mike.Crisp@osd.mil 


Thomas  B.  Blann 
Deputy  Director 
Naval  Warfare 
703-681-5417 
Thomas.Blann@osd.mil 


William  J.  McCarthy 
Deputy  Director 
Net-Centric  &  Space 
Systems 
703-681-5411 
William.McCarthy@osd.mil 


Richard  G.  Sayre 
Deputy  Director 
Live  Fire 
703-614-3991 
Rick.Sayre@osd.mil 


Land  Warfare  Systems 

Expeditionary  Warfare  System 

Future  Combat  Systems 

Battle  Command 

Land  Munitions 

Rotary  &  Tilt-Rotor  Aircraft 

Tactical  UAV 

Chem-Bio  Defense  Program 

Joint  Rapid  Acquisition  Cell 
Integrated  Resource  Analysis 
T&E  Threat  Resource  Activity 
Joint  Test  Board 


Air  Combat  Systems 
Strategic  Air  Combat  Systems 
Aircraft  Carriers 
Air  Launched  Munitions 
Endurance  UAVs 
Air  Warfare  C4ISR  Systems 
IR/UV/RF  Def  Combat  Sys 
Airlift  Systems 

Joint  T&E  Program 
Center  for  Counter  Measures 


Surface  Ships 
Submarines  &  UUVs 
Navy  Combat  Systems 
Radars 
Sonars 
EW  System 
CEC 

Naval  Munitions 
Missiles 

Guided  Projectiles 
Torpedoes 


Space  &  Strategic  Systems 
Net-Centric  Systems 
Strategic  C4ISR  Systems 
Major  Automated  Info  System 
Chemical  Weapons  Demil 

Network  Information 
Assurance  &  Interoperability 
Assessment 


Live  Fire  -  Land,  Air,  Naval 
Lethality  &  Survivability 

Joint  IED  Defeat 
Joint  Live  Fire  Program 
Joint  Aircraft  Survivability 
Joint  Technical  Coordinating 
Group-Munitions 
Effectiveness 
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Why  have  a  Town  Meeting  on  T&E 
role  in  Requirements  Process? 

•  Address  some  of  the  following  commonly 
heard  questions  and  complaints*: 

—  Requirements  are  too  vague  and/or  confusing  to 
test  against. 

—  Where  did  requirement  X,  Y,  or  Z  come  from? 

•  Why  “Speed  X”  versus  “Speed  Y”? 

•  How  does  a  specified  capability  impact  the  mission? 

•  What  happens  to  ability  to  accomplish  the  mission  if  the 
specified  level  of  capability  is  not  achieved? 


*  There  are  probably  others... 


Requirements  Process 

CJCSI  3170.1  Joint  Capabilities  Integration 
and  Development  System  (JCIDS) 

Why  JCIDS? 


•  Support  statutory  requirements  of  JROC  to  validate  and 
Prioritize  Joint  Warfighting  Requirements. 


•  Primary  Objective  is  to  ensure  the  joint  warfighter  receives 
the  capabilities  required  to  successfully  execute  the  missions 
assigned  to  them. 


How  does  JCIDS  work? 


The  JCIDS  Process 
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Strategy 
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with  analysis:  Overlay  what  we 
have  with  what  we  need  to  do. 
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Figure  A-l.  Top  Down  Capability  Need  Identification  Process 


CJCSI  3170.1,  dated  1  May  2007,  page  A-3 


The  JCIDS  Process  cont 

Where  does  T&E  fit  in? 


Capability  Gap  Validation 


Produce  A  Deploy 

JiAetosI 


Where  does  T&E  fit  in? 

According  to  CJCSI  3170.1: 


test  and 


JCIDS  Documentation.  The  documentation  developed  during 
the  JCIDS  process  provides  the  formal  communication  of 
capability  gaps  between  the  operator  and  the  acquisition, 

,  and  resource  management  communities.  The 
document  formats  and  review  processes  specified  in  reference  c 
are  mandatory  and  shall  be  used  throughout  the  DOD  for  all 
acquisition  programs  regardless  of  acquisition  category  (AC AT). 


Question  -  Where  should  T&E  fit  in  the 
Requirements  Development  process? 


Safety  of  Unmanned  Systems 


Sponsored  by 

>nse  Safety  Oversight  Council 
Inquisition- and  Technology 
^Programs  Task  Forceps 
(DSOCATRTF) 

Status  Update 


Mr.  Michael  H.  Demmick 


Agenda 


•  Leadership 

*  Background 

*  Objectives 

*  Approach 

•  Progress 

*  Organization 

•  Workgroup  participants 

*  Precepts  Review 

•  Final  Product 

•  Summary 


Unmanned  Systems 
_ Leadership _ 

•  OSD  Sponsor 

-  Mr.  Mark  Schaeffer,  Director, 
Systems  and  Software  Engineering 
&  Chairman,  DSOC  ATP  TF 

-  Dr.  Liz  Rodriquez-Johnson, 
Executive  Secretary,  DSOC  ATP  TF 


Why  Safety  of  UMSs? 


Rapid  Acquisition  Pace 
Achievable 
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UAV  launch  from 
MDARS 


Talon  Swords 


Raytheon  UCAV 
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UMS  Level  of  Awareness  vs. 
Levels  of  Control 
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Background 


•  In  FY05,  the  OSD  Joint  Robotics 
Program  Coordinator  for  ground 
systems  tasked  Navy  to: 

-  Provide  unifying  safety  guidance  across  all 
ground  robotic  projects 

-  Establish  initial  safety  precepts  for  ground 
robotic  systems 

•  Program  Safety  Guidance 

•  Operational  Guidance 

•  System  Design  Safety  Guidance 

•  Results  briefed  at  2005  ISSC 
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Background 


*  October  2005  briefed  to  OSD  (DSOC  ATP  TF) 

*  ATP  TF  directed  expansion  of  effort  to  include 
all  Unmanned  Systems  (air,  ground,  and  sea) 

*  Emphasized  necessity  of  community  input 

-  Program  Management 

-  Design 

-  Test 

-  Operational 

-  Safety 

*  Emphasized  guidance  vice  direction 
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UMS  Safety  Objectives 


•  Focus  the  technical  community  on  the 
System  Safety  needs  for  UMS 

•  Specifically: 

1.  Understand  the  safety  implications,  including 
legal  issues,  associated  with  the  rapid 
development  and  use  of  a  diverse  family  of 
unmanned  systems  both  within,  and  external  to, 
the  DoD. 

2.  Establish  and  agree  upon  a  standardized  set  of 
safety  precepts  to  guide  the  design,  operation, 
and  programmatic  oversight  of  all  unmanned 
systems. 

3.  Develop  safety  guidance,  such  as  design 
features,  hazard  controls  and  mitigators,  for  the 
design,  development,  and  acquisition  of 
unmanned  systems. 


xM-PIa 


_ Approach _ 

^  Involve  technical  community 

-  Six  Workgroups 

-  Approximately  80  technical  experts 

-  Government,  Industry,  Academia 

Maximize  Community  Awareness 

-  March  2006  Workshop 

•  300  attendees 

-  International  Systems  Safety  Conference  (ISSC) 

-  Association  of  Unmanned  Vehicles  International  (AUVSI) 

-  NDIA  Systems  Engineering  Conference 

^  Obtain  Feedback 

-  Web  Page  (http://www.ih.navy.mil/unmannedsystems) 

-  Tech  Panels  &  Reviews 

s  ISSC  (31  July  -  4  Aug  2006) 

S  AUVSI  (29  -  31  Aug  2006) 

NDIA  Systems  Engineering  (23  -  26  Oct  2006) 

Mr.  Schaeffer’s  Systems  Engineering  Forum 
^  NDIA  Systems  Engineering  (22  -  25  Oct  2007) 


Road  to  Completion 


Held  Three  Workshops 

-  March  2006,  Huntsville 

-  May  2006,  Crystal  City 

-  June  2006,  Crystal  City 

v  Developed  Safety  Precepts 

-  Programmatic  safety  precepts  (6) 

-  Operational  safety  precepts  (5) 

-  Design  safety  precepts  (19) 

Developed  more  detailed  design  safety  “best 
practices”  (safety  precept  clarification  tables) 
(ongoing) 

^ USD  (AT&L)  issued  the  Guide  on  17  July  2007 


Workshop  Organization 

'S  Six  Workgroups 

1.  Precept  Development 

2.  Weapons  Control 

3.  Situational  Awareness 

*  Human-Machine  Interface 

*  Machine-Machine  Interface 

4.  Command  and  Control 

5.  States  and  Modes 

6.  Definitions/Common  Taxonomy 


Unmanned  Systems 
Management  Team 


•  Members 

-  Mr.  Dave  Schulte 

-  Mr.  Ed  Kratovil 

-  Mr.  Jim  Gerber 

-  Ms.  Rhonda  Barnes 

-  Mr.  Danny  Brunson 

-  Mr.  Josh  McNeil 

-  Mr.  Bill  Pottratz 

-  Dr.  Tom  English 

-  Mr.  Steve  Mattern 

-  Mr.  John  Canning 

-  Mr.  Bob  Schmedake 

-  Mr.  Mike  Demmick 
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Special  Thanks 
“Heavy  Lifters” 


S Mr.  Jim  Gerber 
^ Mr.  Mike  Demmick 
S Mr.  Josh  McNeil 
v  Ms.  Rhonda  Barnes 
v  Mr.  Danny  Brunson 


UMS  Safety  Precept  Definitions 


Programmatic  Safety  Precept  (PSP)  =  Program 
management  principles  &  guidance  that  will  help  ensure 
safety  is  adequately  addressed  throughout  the  lifecycle 
process.  (6) 

Operational  Safety  Precept  (OSP)  =  A  safety  precept 
directed  specifically  at  system  operation.  Operational  rules 
that  must  be  adhered  to  during  system  operation.  These 
safety  precepts  may  generate  the  need  for  Design  Safety 
Precepts.  (5) 

Design  Safety  Precept  (DSP)  =  General  design 
guidance  intended  to  facilitate  safety  of  the  system  and 
minimize  hazards.  Safety  design  precepts  are  intended  to 
influence,  but  not  dictate,  specific  design  solutions.  (19) 


Safety  Precepts  for  UMS 


OSD  Policy 


P  M/Operators/ 
User  reps 


PM/Industry 
Design  Team 


Provide  PMs,  designers,  and  systems  safety  managers  with  appropriate  safety 
guidelines  and  best  practices,  while  maintaining  PM’s  flexibility 


Safety  Design  Guidelines 


Are  we  creating  two  sets  of  safety  criteria: 
one  for  manned  systems,  and  one  for  unmanned  systems?? 


Unique  to  Unique  to 

Manned  System  Unmanned  System 
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Safety  Precepts 


^  Did  not  previously  exist 

^  Evolved  through  an  arduous,  but 
thorough,  systems  engineering 
process  over  the  past  2  years 

^  Separate  study  was  performed  to 
determine  if  current  DoD  and/or 
Service-specific  policies  addressed 
each  of  the  safety  precepts 
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Safety  Precepts  (cont’d) 


The  results  of  this  study  indicate: 

S  Safety  precept  PSP-1  is  completely  addressed  in  both  DoD  and 
Service-specific  policies. 

S  Three  precepts  (PSP-4,  PSP-6,  and  DSP-1)  are  completely 

addressed  in  DoD  policy  and  are  partially  addressed  in  Service- 
specific  policies. 

S  Four  precepts  (PSP-3,  DSP-11,  DSP-12,  and  DSP-19)  are  partially 
addressed  in  both  DoD  and  Service-specific  policies. 

✓  Nine  precepts  (PSP-2,  OSP-1,  OSP-3,  OSP-5,  DSP-7,  DSP-13, 
DSP-14,  DSP-16,  DSP-18)  are  not  addressed  in  DoD  policy  but 
are  partially  addressed  in  Service-specific  policy. 

^  Twelve  precepts  (PSP-5,  OSP-2,  OSP-4,  DSP-2,  DSP-4,  DSP-5, 
DSP-6,  DSP-8,  DSP-9,  DSP-10,  DSP-15  and  DSP-17)  are  not 
addressed  in  DoD  nor  Service-specific  policies. 

S  One  precept  DSP-3  was  not  mapped  to  policy. 


Final  Product 

UNMANNED  SYSTEMS  SAFETY  GUIDE  FOR  DOD 

ACQUISITION 
27  June  2007 


Document  contains  descriptive  and 
clarifying  text  for  each  precept. 

Includes  definitions 

But, ...comments/lessons  learned  are 
still  requested  for  future  updates 

-  NOSSA  Website 

(http://www.ih.navy.mil/unmannedsystems) 


USD  (AT&L)  UMS  Memo 


THE,  UNDER  SECRETARY  OF  DEFENSE 

3010  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301  3010 


JUL  1  7  20 D7 


ACQUISITION, 

TTEJCHNOLW'f 
ano  logistics 


MEMORANDUM  I’OR  SECRETARIES  OF  THE  MILITARY  DEPARTMENTS 
CHAIRMAN  OF  THE  JOINT  CHILI  S  OF  STAFF 
UNDER  SECRETARIES  OF  DEFENSE 
COMMANDERS  OF  THE  COMBA  TANT  COMMANDS 
ASSISTANT  SECRETARY  OF  DEFENSE  (NETWORKS  & 
INFORMATION  IN  I  EC, RATION) 

DIRECTOR,  DEFENSE  RESEARCH  AND  ENGINEERING 
DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 
DIRbU  I  OR,  PROGRAM  ANALYSIS  AND  EVALUATION 
DIRECTORS  OF  THE  DEFENSE  AGENCIES 

SUBJECT:  Unmanned  Systems  Safely  Guidance 


In  March  2006,  Hie  Defense  Safety  Oversight  Council  Acquisition  and  Technology 
Programs  Task  Force  (ATP  TF)  initiated  a  study  to  identify  the  unique  safety  challenges 
of  unmanned  systems  (UMSs),  especially  those  systems  carry  ing  and  deploying  weapof 
in  a  joint  environment.  These  safety  challenges  significantly  increase  as  more  IjM^ 
fielded  and  used  in  the  same  warfighting  environment. 

Using  a  collaborative  process  with  experienced  personnel  from  all  Services,  the 
ATPTF  developed  the  “Unmanned  Systems  Safety  Guide  fnrDoD  Acquisition”  to 
provide  programmatic,  operational,  and  design  guidelines  to  support  the  development  and 
fielding  of  safe  UMSs.  Please  you  use  the  Guide,  found  at^  http://www.acq  .osd.mil/atpUy, 
to  help  identify  and  mitigate  hazards  and  their  associated  risks  for  all  UMS  types. 

For  those  UMSs  that  are  ACAT  ID  program,  the  UMS  safety  guidelines  will  be  a 
special  interest  item  during  OSD  Program  Support  Reviews.  U MS-spccific  guidelines 
have  been  added  to  the  Defense  Acquisition  Program  Support  methodology  to  guide  the 
evaluation  of  how  successfully  programs  have  engineered  UMSs  to  Teducc  safety  risks  to 
acceptable  levels. 


use  the  Guide  to 
help  identify  and 
mitigate  hazards 
and  their  associated 
risks  for  all  UMS  types.” 


—Kcnneih  J 


Guidelines  are  a 
special  interest  item 
during  OSD 
Program  Support 
Reviews  for  ACAT 
ID  UMS  Programs. 
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Programmatic  Safety  Precepts 


PSP-1*:  The  Program  Office  shall  establish  and  maintain  a  system 
safety  program  (SSP)  consistent  with  MIL-STD-882. 

PSP-2*:  The  Program  Office  shall  establish  unifying  safety 

precepts  and  processes  for  all  programs  under  their 
cognizance  to  ensure: 

-  Safety  consistent  with  mission  requirements,  cost  and 
schedule 

-  Mishap  risk  is  identified,  mitigated  and  accepted. 

-  Each  system  can  be  safely  used  in  a  combined  and 
joint  environment 

-  That  all  safety  regulations,  laws,  and  requirements  are 
met. 

PSP-3*:  The  Program  Office  shall  ensure  that  off-the-shelf  items 
(e.g.,  COTS,  GOTS,  NDI),  re-use  items,  original  use  items, 
design  changes,  technology  refresh,  and  technology 
upgrades  (hardware  and  software)  are  assessed  for 
safety,  within  the  system. 


Programmatic  Safety  Precepts 

(Cont’d) 


PSP-4*:  The  Program  Office  shall  ensure  that  safety  is 
addressed  for  all  life  cycle  phases. 

PSP-5:  Compliance  to  and  deviation  from  the  safety  precepts 
shall  be  addressed  during  all  Milestone 
decisions  and  formal  design  reviews  such  as  System 
Requirements  Review  (SRR),  Preliminary  Design 
Review  (PDR),  and  Critical  Design  Review  (CDR). 

PSP-6*:  The  Program  Office  shall  ensure  UMS  designs  comply 
with  current  safety  and  performance  criteria. 


Note:  While  the  document  serves  only  as  a  guide,  usage  of  the  terms 
“shall”  and  “should”  reflects  the  level  of  concern  of  the  safety 
community 

*  Denotes  applicability  to  both  manned  and  unmanned  systems. 


Operational  Safety  Precepts 


OSP-1:  The  controlling  entity(ies)  of  the  UMS  should  have 
adequate  mission  information  to  support  safe 
operations. 

OPS-2:  The  UMS  shall  be  considered  unsafe  until  a  safe  state 
can  be  verified. 

OPS-3:  The  authorized  entity(ies)  of  the  UMS  shall  verify  the 
state  of  the  UMS,  to  ensure  a  safe  state  prior  to 
performing  any  operations  or  tasks. 

OSP-4*:  The  UMS  weapons  should  be  loaded  and/or  energized 
as  late  as  possible  in  the  operational  sequence. 

OSP-5*:  Only  authorized,  qualified  and  trained  personnel,  with 

the  commensurate  skills  and  expertise  using  authorized 
procedures,  shall  operate  or  maintain  the  UMS. 
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Design  Safety  Precepts 


DSP-1*:  The  UMS  shall  be  designed  to  minimize  the  mishap  risk  during 
all  life  cycles  phases. 

DSP-2:  The  UMS  shall  be  designed  to  only  respond  to  fulfill  valid 
commands  from  the  authorized  entity(s). 

DSP-3:  The  UMS  shall  be  designed  to  provide  information,  intelligence, 
and  method  of  control  (I2C)  to  support  safe  operations. 

DSP-4*:  The  UMS  shall  be  designed  to  isolate  power  until  as  late  in  the 
operational  sequence  as  practical  from  items  such  as:  a) 
Weapons,  b)  Rocket  motor  initiation  circuits,  c)  Bomb  release 
racks,  or  d)  Propulsion  systems. 

DSP-5*:  The  UMS  shall  be  designed  to  prevent  release  and/or  firing  of 
weapons  into  the  UMS  structure  or  other  weapons. 

DSP-6*:  The  UMS  shall  be  designed  to  prevent  uncommanded  fire  and/or 
release  of  weapons  or  propagation  and/or  radiation  of 
hazardous  energy. 

DSP-7*:  The  UMS  shall  be  designed  to  safely  initialize  in  the  intended 
state,  safely  and  verifiably  change  modes  and  states,  and 
prevent  hazardous  system  mode  combinations  or  transitions. 


Design  Safety  Precepts 

(Cont’d) 


DSP-8*:  The  UMS  shall  be  designed  to  provide  for  an 

authorized  entity(s)  to  abort  operations  and  return  the 
system  to  a  safe  state,  if  possible. 

DSP-9*:  Safety  critical  software  for  the  UMS  design  shall  only 
include  required  and  intended  functionality. 

DSP-10*:  The  UMS  shall  be  designed  to  minimize  single¬ 
point,  common  mode  or  common  cause  failures 
that  result  in  high  and/or  serious  risks. 

DSP-11*:  The  UMS  shall  be  designed  to  minimize  the  use 
of  hazardous  materials. 

DSP-12*:  The  UMS  shall  be  designed  to  minimize 
exposure  of  personnel,  ordnance,  and 
equipment  to  hazards  generated  by  the  UMS 
equipment. 

DSP-13*:  The  UMS  shall  be  designed  to  identify  to  the 
authorized  entity(ies)  the  weapon  being 
released  or  fired,  but  prior  to  weapon  release  or  fire. 


Design  Safety  Precepts 

(Cont’d) 


DSP-14*:  In  the  event  of  unexpected  loss  or  corruption  of 
command  link,  the  UMS  shall  transition  to  a  pre¬ 
determined  and  expected  state  and  mode. 

DSP-15*:  The  firing  of  weapons  systems  shall  require  a 

minimum  of  two  independent  and  unique  validated 
messages  in  the  proper  sequence  from  the  authorized 
entity(ies),  each  of  which  shall  be  generated  as  a 
consequence  of  separate  authorized  entity  action. 
Both  messages  should  not  originate  within  the  UMS 
launching  platform. 

DSP-16:  The  UMS  shall  be  designed  to  provide  contingencies 
in  the  event  of  safety  critical  failures  or  emergencies 
involving  the  UMS. 

DSP-17:  The  UMS  shall  be  designed  to  ensure  safe  recovery  of 
the  UMS. 

DSP-18*:  The  UMS  shall  ensure  compatibility  with  the  test  range 
environment  to  provide  safety  during  test  and 
evaluation. 

DSP-19*  The  UMS  shall  be  designed  to  safely  operate  within 
combined  and  joint  operational  environments. 
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Precept  Clarification  Table 


Precept  Number:  Statement  of  the  precept  in  the  form  of  a 
requirement  or  general  guidance. 

Scope:  Answers  the  question  of  “What?”  the  precept  is  for;  often 
can  be  answered  by  “This  precept  addresses....” 

Rationale:  Answers  the  question  of  “Why?”  the  precept  is  required. 
This  provides  addition  clarification  of  the  intent  of  the  precept. 

Example:  Provide  as  many  clarifying  explicit/real-world  examples  to 
demonstrate  the  issues  and  specific  hazards  the  precept  addresses. 

Detailed  Considerations:  Answers  the  question  of  “How?”  by 
providing  details  to  assist  with  implementation  of  the  precept.  These 
are  specific  statements  written  in  the  form  of  a  requirement  or 
guideline  which  capture  lessons  learned  and  experience  from  other 
programs.  Some  of  these  considerations  can  be  tailored  for  specific 
programs  and  incorporated  into  system  specifications  as  safety 
requirements. 
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DSP-19  Combined  Joint 
Operational  Environments 

DSP-19*  The  UMS  shall  be  designed  to  safely  operate  within 
combined  and  joint  operational  environments. 

Scope:  The  intent  of  this  precept  is  to  consider  interoperability 
of  the  UMS  with  manned  systems  (unmanned  undersea  systems 
with  ships,  UAVs  with  manned  military  or  commercial  aircraft). 
This  precept  addresses  de-confliction  of  air  corridors  and  use  of 
UMSs  for  non-military  peace-time  operations  such  as  disaster 
relief  and  boarder  patrol.  This  also  addresses  potential  ad-hoc 
combinations  of  systems  by  the  field  commander(s)  that  may  not 
have  originally  intended  to  operate  as  combined  systems  or  as 

an  SoS. 

Rationale:  The  intent  of  this  precept  is  to  provide  safety 
compatibility  among  independently  developed  systems  operating 
in  a  combined  or  joint  operational  environment 


DSP-19  Combined  Joint 
Operational  Environments  (contd) 


DSP-19*  The  UMS  shall  be  designed  to  safely  operate  within 
combined  and  joint  operational  environments. 

Examples: 

1.  Use  of  a  UAV  within  the  National  Air  Space  (NAS). 

2.  Use  of  UMSs  for  non-military  peace-time  operations  such  as 
disaster  relief  and  border  patrol. 

3.  Multiple  UMSs,  operating  in  a  net-centric  environment,  could 
tax  the  communications  network  bandwidth. 
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DSP-19  Combined  Joint 
Operational  Environments  (contd) 

DSP-19*  The  UMS  shall  be  designed  to  safely  operate  within 

combined  and  joint  operational  environments. 

Detailed  Considerations: 

•  Communication  reliability,  network  availability/quality  of  service 
and  data/information  assurance  shall  be  commensurate  with  the 
safety  criticality  of  the  functions  supported. 

•  The  system  should  be  designed  to  be  operated  and  transported 
during  non-wartime  conditions  within  normal  transportation  and 
commercial  airspace  environments  meeting  the  requirements  of 
the  DOT,  FAA,  ETS  300-019,  Part  1-2,  IEC  721. 

•  Reference  the  Society  of  Automotive  Engineers  AS-4  Joint 
Architecture  for  Unmanned  Systems  (JAUS)  Working  Group  related 
to  interoperability. 

•  In  accordance  with  CJCSI  3170  directives,  all  systems  will  be 
reviewed  for  safety  within  the  joint,  combined,  and  SoS 
environments. 

•  Reference  NATO  STANAG  4586  Section  1.1  (para  6). 


DSP-19  Combined  Joint 
Operational  Environments 


DSP-19*  The  UMS  shall  be  designed  to  safely  operate  within 
combined  and  joint  operational  environments. 

Existing  Policy: 


Existing  Policy: 

Service 

Document 

Section 

Comment 

Joint  Chiefs  of  Staff 

DoD  CJCSI  3170.01  E 

Text  implies 

precept. 

DoDD 

4630.0544.1 

Text  implies 
precept. 

Need  your  help  in  identifying  any  other  existing 

policy  documents 


Summary 

Held  three  workshops  (March,  May,  June  2006) 

S  Government/industry/academia  teams  developed  draft  safety 
precepts,  rationale  &  design  guidance 

>  All  Services  and  numerous  UMS  program  office  reps 
participating 

S  Developed  Complex  Programmatic,  Operational,  and  Design 
safety  precepts 

S  Briefed 

>  International  Systems  Safety  Conference  (2005,  2006  and 
2007) 

>  AUVSI  (August  2006) 

>  NDIA  Systems  Engineering  (October  2006  and  2007) 

>  NDIA  24th  Annual  National  Test  &  Evaluation  Conference 
(February  2008) 

S  Comments  Requested 

>  NOSSA  Website  (http://www.ih.navy.mil/unmannedsystems) 


Summary  (cont’d) 

USD  (AT&L)  Memorandum  of  17  July  2007 

'S  Forwarded  the  Guide  to  the  Service  Secretaries  and  other  major 
DoD  components  strongly  endorsing  it  for  all  UMS  acquisitions. 

v  Undersecretary  directed  that  the  UMS  Safety  precepts  in  the 
Guide  be  a  special  interest  item  for  ACAT  ID  Program  Reviews. 

S  The  Guide  has  been  posted  on  the  OSD  ATP-TF  Website  at 
http://www.acq.osd.mil/atptf/ 

S  Next  steps: 

^  Convert  the  Guide  to  a  MIL-HDBK 

•  Institutionalizes  guidance,  and  facilitates  Service 
ownership 

•  Formatting  complete;  final  Handbook  estimated  3rd  Qtr 
2008 

^Update  Policy  and  Service  Directives  to  address  UMS 
Precepts,  where  appropriate.  (Remember,  12  Safety 
Precepts  not  addressed  at  all  in  policy.) 
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Contact  Information 
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Name:  Michael  H.  Demmick 

Phone:  (301)744-4932 
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E-Mail:  Michael.Demmick@navv.mil 
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DSP-14  Loss  of  Command  Link 

DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of 
command  link,  the  UMS  shall  transition  to  a  pre-determined  and 
expected  state  and  mode. 

Scope:  This  precept  addresses  the  overall  UMS  design  architecture  and 
states  and  mode  management  in  the  event  of  unexpected  loss 
or  corruption  of  the  command,  control,  and  communications  link  (i.e. 
loss  of  data  link,  loss  of  command  and  control).  The  objective  is  for  the 
UMS  to  be  in  the  anticipated/expected  state  when  recovery  occurs.  It  is 
not  the  intended  communication  loss  as  in  the  case  of  underwater 
vessels  or  other  fully  autonomous  UMS.  The  system  should  have  the 
capability  of  storing  a  set  of  actions  to  take,  or  states  to  transition  to, 
when  the  command  link  is  lost.  Predetermined  means  we  have  them  in 
the  plan.  Expected  means  we  intend  that  portion  of  the  plan  to  go  into 
effect  for  this  condition.  It  applies  to  both  the  test  and  perational 
environments.  This  precept  is  related  to  DSP-3  and  DSP-16. 

Rationale:  The  intent  of  this  precept  is  to  assure  that,  by  design;  the 
controlling  entity  can  anticipate  the  status,  mode  and  state  of  the 

UMS,  and  any  on-board  weapons  during  a  loss  of  link  period,  corruption 
of  link,  and  the  subsequent  recovery  of  link.  Determination  of  pre¬ 
determined  and  expected  status  should  be  based  on  analysis  of  such 
things  as  CONOPS,  mission  profile,  and  threat  hazard  assessments. 

DSP-14  Loss  of  Command  Link 

(cont’d) 


DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of 
command  link,  the  UMS  shall  transition  to  a  pre-determined 
and  expected  state  and  mode. 

Examples: 

1.  A  UAV  would  continue  to  fly  out  of  range  upon  loss  of 
command  link  if  no  contingency  provisions  are  designed  into 
the  system. 

2.  A  UAV  has  been  directed  upon  loss  of  link  to  return  to  base. 
It  currently  has  mission  parameters  loaded,  weapons  have  been 
energized,  and  commanded  to  fire  when  communications  link 
has  been  lost.  The  UAV  responds  to  its  mission  parameters 
and  is  returning  to  base  when  it  re-establishes 
communications.. ..what  state  are  the  weapons  in?  Will  it  now 
execute  its  command  to  fire?  If  communications  are  lost  and 
re-established,  the  UAV  and  weapons  should  default  to  an 
expected  state. 


DSP-14  Loss  of  Command  Link 

(cont’d) 


DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of  command 
link,  the  UMS  shall  transition  to  a  pre-determined  and  expected 
state  and  mode. 


Detailed  Considerations: 

•  The  design  should  define  state  and  mode  transitions,  including 
a  desired  and/or  predictable  course  of  action  (such  as  move 
physically  to  a  safe  zone  or  crash  in  a  safe  zone),  in  the  event 
of  loss  of  link  or  intermittent  command  and  control.  The  criteria 
for  pre-determined  and  expected  states  and  modes,  and  the 
courses  of  action  include: 

-the  UMS  CONOPS  and  application; 

-  the  level  of  autonomy  and  level  of  control; 

-  the  operating  environment  (i.e.  training,  test,  underwater, 
airborne,  etc.); 

-  the  adequacy  of  communication  link. 


DSP-14  Loss  of  Command  Link 
_ (cont’d) _ 

DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of  command 
link,  the  UMS  shall  transition  to  a  pre-determined  and  expected 
state  and  mode. 

Detailed  Considerations:  (cont’d) 

The  UMS  design  should  consider  retention  of  pertinent  mission 
information  (such  as  last  known  state  and  configuration,  etc.) 
for  the  UMS  and  the  controlling  entity(ies)  to  recover  from  loss 
of  the  communications  link. 

•  The  UMS  design  must  consider  limiting  the  duration  for  which 
undelivered  messages  are  considered  valid. 

•  The  UMS  design  must  consider  undelivered  messages  that  can 
exist  within  the  communication  system. 

•  The  UMS  should  ensure  command  messages  are  prioritized  and 
processed  in  the  correct  sequence  and  in  the  intended  state  and 
mode. 

•  Reference  NATO  STANAG  4404  Section  7.4  and  8.3.  DoD  8500.1 
Section  4.1;  and  DoD  5000.1  Section  El. 1.9. 


DSP-14  Loss  of  Command  Link 
_ (cont’d) _ 

DSP-14*  In  the  event  of  unexpected  loss  or  corruption  of  command 
link,  the  UMS  shall  transition  to  a  pre-determined  and  expected 
state  and  mode. 
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Background 


7 YSR/N 


•  The  Time  Space  Position  Information  (TSPI)  T&E 
Instrumentation  Technology  Investment  Plan. 

•  A  Defense  Test  Resources  Management  Center  (DTRMC) 
initiative  within  the  Central  Test  and  Evaluation  Investment 
Program  (CTEIP). 

-  Started  as  a  GPS  TSPI  Technology  Investment  Plan  several  years  back. 

•  Tri-Service  test  ranges  provided  a  significant  amount  of  data  that 
has  been  compiled  regarding  current  TSPI  capabilities  versus 
projected  requirements. 

•  Several  “Common  Test  &  Training  Range  Architecture” 
(CTTRA)  Workshops  were  conducted  with  government  and 
private  industry  to  collect  information  as  well. 


Background 


7 YSR/N 


•  A  draft  document  titled  -  “TSPI  Capabilities  Survey ,  Needs 
Assessment  and  Preliminary  Gap  Analysis ”  was  produced  with 
the  data. 

-  The  objective  was  to  identify  current  TSPI  capabilities  at  the  Major 
Range  Test  and  Facility  Base  (MRTFB)  facilities  versus  emerging  TSPI 
needs. 

-  Capture  the  deltas  between  current  capabilities  and  emerging 
requirements. 

-  Provide  some  insight  into  emerging  technologies  that  could  potentially 
bridge  the  gap. 

•  In  essence,  provide  a  roadmap  for  future  CTEIP  investment 
programs  involving  (originally)  GPS  TSPI  instrumentation. 


Background 


7 YSR/N 


•  CTEIP  determined  there  were  significant  gaps  and  holes  in  the 
data  collected  after  reviewing  the  TSPI  Technology  Investment 
Plan  (TSPI  TIP). 

•  The  Range  Commanders  Council’s  (RCC)  Electronic  Trajectory 
Measurement  Group  (ETMG)  was  approached  to  review  and 
update  the  TSPI  TIP. 

•  Identify  the  holes  in  the  current  TSPI  capabilities  section  and  finish 
filling  them  in. 

•  Scrub  the  emerging  requirements  and  update  them. 

•  Update  and  validate  the  gap  analysis  between  current  TSPI  capabilities 
versus  future  TSPI  requirements. 

•  Identify  emerging  technologies  that  could  potentially  be  used  to  bridge 
the  gap. 
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Initial  Evaluation 


•  An  analysis  of  the  TSPI  TIP  determined: 

-  The  TIP  spent  a  considerable  amount  of  effort  analyzing  and  documenting 
all  TSPI  capabilities  (radar,  GPS,  optics,  acoustics,  multilateration,  etc.),  not 
just  GPS  TSPI  capabilities. 

-  The  ETMG  felt  the  gap  analysis  focused  on  high  level  requirements  that 
didn’t  really  provide  any  useful  level  of  detail. 

-  It  also  utilized  survey  responses  from  the  T&E  community  which  are  useful, 
but  not  always  accurate. 

•  Hard  to  know  if  the  person  filling  out  the  survey  was  the  person  with  the 
right  knowledge  base  and  background. 

•  Filling  out  surveys  tends  to  be  low  on  everyone’s  priority  list. 

•  The  predominant  theme  tended  to  lean  toward  obsolescence  versus 
inability  to  meet  “current”  TSPI  requirements. 
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TSPI  TIP  Update 


WBRIN 


•  The  ETMG  asked  these  questions  after  reviewing  the  TSPI 
TIP: 

-  How  can  we  take  what  has  been  collected  thus  far  and  turn  it  into  a 
useful  tool  for  CTEIP? 

-  With  so  much  effort  put  into  documenting  all  TSPI  instrumentation’s 
current  capabilities,  should  the  gap  analysis  focus  on  just  GPS 
instrumentation  shortfalls  or  should  it  focus  on  TSPI  capability 
shortfalls  in  general  regardless  of  the  type  of  instrumentation  involved? 

-  How  do  we  validate  the  future  TSPI  requirements  in  order  to  conduct  a 
meaningful  gap  analysis  of  current  capabilities  versus  future 
requirements? 

-  Who  really  should  be  providing  these  future  TSPI  requirements  -  i.e. 
the  MRTFB  ranges,  their  customers,  or  a  combination  of  both? 

-  Should  an  attempt  be  made  to  identify  emerging  technology  that  could 
hold  the  potential  to  meeting  the  identified  future  requirements? 


6 


TSPI  TIP  Update 


WBRIN 


•  The  approach  adopted  to  update  the  TSPI  TIP  included  the 
following: 

-  Start  with  filling  in  the  holes  in  the  MRTFB  current  TSPI  capabilities 
section  as  well  as  identify  the  shortfalls  in  meeting  current  TSPI 
requirements  from  a  “Range”  perspective. 

-  Don’t  use  surveys  but  rather  contact  the  ranges  either  by  phone  or  face  to 
face,  and  talk  to  the  owners/operators  of  the  TSPI  hardware. 

-  Obtain  a  much  more  accurate  inventory  of  current  TSPI  capabilities. 

-  Talk  to  the  test  managers  and  get  their  perspective  on  whether  or  not  they 
can  meet  their  customers  TSPI  requirements  of  today. 

-  Identify  who  their  top  5-7  customers  are  today  (more  on  this  later!) 

-  Get  the  range’s  perspective  on  where  they  see  TSPI  requirements  (for  all  the 
types  of  testing  they  do)  headed  over  the  next  5-10  years  without  tying  it  to 
specific  instrumentation. 


TSPI  TIP  Update 


WBRIN 


•  Identify  and  document  (to  the  extent  possible)  known  and 
projected  TSPI  requirements  5-10  years  from  now. 

-  Very  tough  to  do! 

-  Identify  the  types  of  TSPI  required  including  performance  expectations, 
types  of  data  required,  level  of  accuracy/resolution,  real-time  or  post 
processed,  regardless  of  the  type  of  TSPI  instrumentation  that  would 
potentially  provide  this  data. 

-  More  on  this  later! 

•  Review  current  capabilities  (all  TSPI  instrumentation)  versus 
known  or  projected  future  TSPI  requirements  and  conduct  a  gap 
analysis. 

•  Document  shortfalls  where  possible  (known  or  projected). 


8 


Identifying  Emerging 
TSPI  Requirements 


7 YSR/N 


•  Considered  to  be  the  toughest  challenge  in  updating  the  TSPI  TIP. 

•  Identifying  “real”  hard  requirements  versus  “I  think  this  will  be 
the  requirements”  or  the  notion  of  “If  we  build  it,  they  will  come” 
type  of  requirements  is  hard  to  do. 

•  Obtaining  funding  for  new  range  instrumentation  or  upgrades  to 
existing  instrumentation  is  hard  to  come  by. 

•  Approving  a  new  I&M  or  CTEIP  program  based  on  the  two 
reasons  listed  above  is  not  going  to  fly. 

•  To  the  extent  possible,  we  need  documented  requirements  from 
range  customers. 

-  What  systems  will  be  tested  at  the  MRTFB  ranges  5-10  years  from  now? 

-  What  types  of  TSPI  is  going  to  be  need  and  at  what  level  of 
accuracy/resolution? 


7 YSR/N 


Identifying  Emerging 
TSPI  Requirements 


•  Problems  faced  in  obtaining  hard  data  on  emerging  TSPI 
requirements: 

-  Obtaining  input  directly  from  the  range  test  mangers  and  equipment 
operators  is  a  key  part  of  the  process  but  not  the  only  component. 

•  Their  perspective  tends  to  be  limited  to  what  types  of  TSPI 
requirements  they  have  today  as  well  as  looking  at  the  progression 
of  TSPI  requirements  over  recent  years  in  order  to  project  future 
requirements. 

-  The  more  important  component  in  identifying  future  TSPI  requirements 
has  to  come  from  the  program  offices,  companies  and  vendors  (i.e. 
range  customers)  that  use  the  MRTFB  ranges  to  conduct  testing 
requiring  TSPI  data. 
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7 YSR/N 


Identifying  Emerging 
TSPI  Requirements 


•  MRTBF  range  customers  -  government  program  offices,  hardware 
manufactures,  etc.  hold  the  key  to  identifying  future  TSPI 
requirements. 

-  We  must  find  out  from  them  what  hardware,  equipment,  or  systems  they 
plan  on  bringing  to  our  MRTFB  ranges  in  the  next  5-10  years  to  test. 

-  We  must  identify  their  TSPI  requirements  in  order  to  successfully  test  these 
systems. 

•  This  can  only  be  accomplished  if  we  know  what  their  test 
requirements  are. 

-  What  are  the  performance  specifications  they  have  to  test  and  validate? 

-  What  are  the  types  of  test  scenarios  they  will  employ? 
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7 YSR/N 


Identifying  Emerging 
TSPI  Requirements 


•  Problems  faced  by  many  programs  that  use  MRTBF  ranges  for 
testing: 

-  They  do  not  usually  consider  specific  TSPI  requirements  they  have  early 
on  in  their  programs. 

-  They  focus  is  on  hardware  and  system  development  and  identifying  their 
own  performance  specifications. 

-  Quite  often  very  little  thought  is  given  to  what  kind  of  TSPI 
instrumentation  and  data  they  are  going  to  need  to  validate  their  system 
performance  until  they  get  close  to  being  ready  to  utilize  a  test  range. 

-  Many  times,  they  do  not  have  a  good  understanding  of  the  TSPI 
capabilities  available  in  order  to  determine  if  they  can  adequately  test  their 
system  in  the  test  scenarios  that  need  to  be  conducted. 
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Documenting  Emerging 
TSPI  Requirements 


•  The  ETMG  will  attempt  to  contact  as  many  of  the  program  offices 
and  companies  as  possible  after  identifying  the  top  5-7  customers 
for  each  T&E  range. 

-  Discuss  what  they  have  on  the  drawing  board  (new  systems,  upgrades  to 
existing  systems,  etc). 

-  Determine  what  types  of  testing  will  they  expect  to  conduct  at  the  T&E 
ranges. 

-  Identify  the  system  performance  requirements  they  will  have  to  validate 
through  testing  such  that  we  can  back  our  way  into  what  TSPI 
instrumentation  will  be  required  and  the  level  of  accuracy/resolution 
needed. 

-  Where  possible,  have  the  various  customers  submit  these  requirements  in 
writing. 
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TfBJRiNa  Customer  Understanding 

Of  TSPI  Capabilities 


•  The  ETMG  will  discuss  current  TSPI  capabilities  with  customers. 

-  Try  and  determine  what  the  customers’  level  of  understanding  is  about 
TSPI  capabilities  at  the  ranges. 

-  Educate  them  where  needed  on  the  current  TSPI  capabilities  available  and 
help  them  determine  if  what  we  have  now  is  going  to  meet  their 
requirements  5-10  years  from  now. 

•  But  you  may  ask,  if  they  are  already  T&E  range  customers,  they 
should  know  what  capabilities  are  there! 

•  Surprisingly  this  is  not  necessarily  the  case  (e.g.,  Boeing) 
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Identify  Emerging 
Technology 


7 YSR/N 


•  The  final  phase  of  updating  the  TSPI  TIP  will  be  identifying 
emerging  technology  that  holds  the  potential  for  meeting  future 
TSPI  requirements. 

•  This  will  be  a  high  level  look  at  emerging  technology  due  to  the 
time  constraints  for  completing  this  document. 

-  The  ETMG  feels  that  identifying  and  documenting  future  TSPI 

requirements  and  conducting  a  gap  analysis  is  the  most  important  part  of 
this  task. 

•  Current  TSPI  instrumentation  capabilities  and  associated 
technology  will  be  compared  to  emerging  technology  and 
recommendations  will  be  provided  on  potential  technology  areas 
to  explore  in  the  future. 
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Summary 


7 YSR/N 


•  The  ETMG  will  update  and  validate  the  existing  TSPI  Capabilities 
Survey,  Needs  Assessment  and  Preliminary  Gap  Analysis. 

•  Update  current  existing  TSPI  instrumentation  capabilities. 

•  Focus  on  identifying  emerging  TSPI  requirements. 

-  Obtain  formal  documentation  on  these  new  requirements  when  possible. 

•  Perform  a  gap  analysis  on  existing  TSPI  capabilities  versus  future 
TSPI  requirements. 

•  Update  the  TSPI  Technology  Investment  Plan  based  on  the  gap 
analysis 


Provide  CTEIP  with  a  viable  planning  tool  they  can  use  for  years 
to  come. 
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Creates  a  huge  spectrum  of  solutions 


Why  is  it  hard  for  UASs  to  fit  in  the 
Manned  Aircraft  T&E  Community? 


Manned  Aircraft 


100  years  of 
refining  processes 
and  thinking 

'Set  communities 

•  Strike 

•  Rotary  wing 

•  Etc 

'Standard  Testing  areas 
'  Flying  qualities 

•  Mission  systems 
'  Propulsion 

•  Etc 


UASs 


-15  years  of 
rushing  to  field 

'  Components 

•  Air  Vehicles 

'  Control  Stations 

•  Links 

'  Sensors 
•  Types 
'  Micro 

•  Mini 

'  Small 

•  Tactical 

•  Strategic 


Lets  make  sure  we  don’t  try  to  fit  a  square  peg  in  a  round  hole 


UAS  T&E  Requirements 


Manned  aircraft  T&E  requirements  well  known 
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- Largely  Unknown - 

Across  broad  spectrum  of  UAS  T&E 


Raven 

4-ft  Wingspan 

A 1  ^ 

All  Sizes,  Capabilities,  &  Costs 


Global  Hawk 
131 -ft  Wingspan 


Maximum  Payload  Weight  (lbs) 


Broad  Spectrum  of  Designs, 
Capabilities,  &  Missions 


Wide  variety  of  capabilities 
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UAS  Categories  by  Payload  and  Radius 


BUSTERO^ 

Raven  O 

Dragon 

Hornet  / 

-oo^  4 

pS  -  Strategic 

Strategic  UAS 
<40,000  AGL 

pO  -  Operational 

Operational  UAS 
<40,000  AGL 


ATCA  FPASS  /  \  0 
1  Pointer  1  u 


100  1,000  10,000 

Maximum  Radius  -C> 


T3  -  Tactical  3 

•  Conventional  / 
VTOL 

•Tactical  UAS 
•<10,000  AGL 


oT2  -  Tactical  2 

•  Mobile  launch 
•Tactical  UAS 
•<5,000  AGL 

°T1  -  tactical  1 

•  Hand  launch 
•Tactical  UAS 
•<1,000  AGL 


Note:  BAMS  is  not  shown  as  final  threshold  max  payload  and  mission  radius  are  still  TBD 


Technical  Challenges 


•  Wide  variety  of  sizes,  performance,  capabilities, 
and  costs 

•  Most  hardware  and  software  proprietary 

*  Non-standard 

*  Inhibits  interoperability 

*  Increases  costs 

•  Airspace  integration 

*  Not  designed  to  operate  in  National  Airspace  System 
(NAS) 

*  Difficult  to  obtain  Certificate  of  Airworthiness  and 
FAA  and  DOD  flight  clearances 

*  Lack  of  Identify  Friend  or  Foe  (IFF)  and 
other  standard  equipment 


Technical  Challenges 
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Range  Challenges 


Each  System  Has  Different  Scope 
of  Requirements 


Different: 

•  Concepts  of  Operations  for 
interoperability  and  airspace  integration 

•  Frequency  de-confliction 

•  Encroachment 

•  Vast  array  of  physical  problems 


Safety  is  a  major  driver 


Social  Challenges 


Different  perceptions 

*  Pilot  off-board  vs.  onboard 

*  Risk  and  consequences  of  UAS  mishaps 

*  Manned  vs.  unmanned  testing  standards 

Risk  management 

*  Commercial,  Range,  and  Military  operations  differ 

*  Attritable  systems  still  have  risk  and  impact 

*  Trade-space  differs  from  manned  testing 

*  Lost-link  procedures 


•  H.R.  2881  FAA  Reauthorization  Act 

•  Integrated  frequency  de-confliction  system 

•  Reallocation  of  1400  MHz  of  new  UAS  flight 
test  bands 

•  Adapting  processes,  facilities,  and  procedures 

*  Updating  Range  Safety  and  Air  Operations  manuals 

*  Better  review  and  analysis  of  UAS  technologies 
and  capabilities 

*  Increasing  Use  of  M&S/ground  test  facilities 


Successfully  and  safely 
integrated  Global  Hawk  and 
manned  aircraft  operations 


Expanded  use  of  outlying 
airfields  and  Range  Facilities 


Air  routes  are 
being  established 
and  exploited 
today 
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Flight  Routing 
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(conclusions 


•  Broad  spectrum  of  systems 

*  Unique  set  of  Technical,  Range,  and  Social 
Challenges 

•  UAS  Test  &  Operation  not  standardized 

*  Doesn't  always  fit  manned  aviation  construct 

•  Processes  and  infrastructure  need  to  catch  up 

•  AirOps,  Range  Safety,  airspace,  communications, 
runways,  hangars 

•  UASs  represent  great  new  potential 

•  Many  successes  starting  to  leverage  these 
capabilities 


UASs  are  revolutionizing  the  way  we  prepare 
for  and  fight  wars  in  the  21st  Century 


Jeremy  Dusina 


Juan  Carlos  Sanchez 


Improving  Requirements 
&  Testing  Process  for 
Rapidly  Fielded 
Unmanned 
Aerial 


Systems 


Army  Test  and  Evaluation  Command 


\\ 


Outline 


*  Unmanned  Aircraft  Systems 

-  ATEC  Mission  Statement 

-  Test  &  Evaluation 


challenges 

-  Recommendations 

-  Conclusions 


Provide  the  warfighter  adequately  tested 
systems  with  robust  evaluations  in  a 

timely  manner  to  support  Rapid  fielding 
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ATEC  Mission  Statement 


•  Facilitate  equipment  procurement/fielding 
decisions  through  testing  and  analysis  to  ensure 
our  Army’s  Warfighters  have  the  right 
capabilities  for  success  across  the  entire 
spectrum  of  operations. 

•  Conduct  rapid  testing  in  direct  support  of  the 
QWOT  warfighter,  providing  capabilities  and 
limitations  analyses  of  weapon  systems  to 
enable  employment  decisions  for  rapid  fielding 
to  the  Combat  Soldier. 


4 


T&E  Challenges  -  Rapid 

Fielding 

•  ONS  lack  extensive  operational  requirements 

-  Mission  not  defined  until  system  is  deployed 

-  Many  times,  difficult  to  quantify 

•  Limited  test  schedule  and  scope: 

-  Use  prior  test  data  when  possible 

-  Leverage  existing  training/exercises 

•  Warfighers  want  systems  because  they 
are  ‘better  than  nothing’ 

-  Must  still  be  tested  to  determine 
capabilities  and  limitations  of  the  system 

•  No  key  denominator  for  UAS 


T&E  Challenges  - 
Capabilities  &  Limitations 

•  Include  aviation  and  infantry  based  requirements 
and  operations 

•  Must  develop  a  robust  list  of  capabilities  and 
limitations  of  the  system  for  the  Warfighter: 

-  Provides  the  warfighter  with  the  intended  operating 
envelope/environment  of  the  system 

•  Rapid  systems  provide  users  with  added  capability 

-  Do  not  replace/improve  existing  equipment 

•  Minimal  analysis  has  not  provided  the  Army  a  clear 
picture  of  how  best  to  employ  UAS 


Recommendations 


•  Simplified  Rapid  Acquisition  Process 

•  Must  have  early  TRADOC  involvement  with  users/units, 
PM,  and  ATEC  for  rapid  equipping 


*TCM  -  TRADOC  Capabilities  Manager 
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Conclusions 


•  UAS  is  a  rapidly  growing  field  within  the  Army 

•  OIF  and  OEF  require  systems  to  be  fielded  as 
quick  as  possible  to  support  the  warfighter 

•  Need  for  improved  rapid  acquisition  process 

•  T&E  Community  will  be  able  to  support  user  with 
a  system  that  will  better  meet  mission  needs 

•  Difficult  to  Quantify 

-  Force  Multiplier 

-  Situational  Awareness 

•  Does  the  Warfighter  need  all  of  these  systems? 

•  What  is  the  key  denominator? 
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Contact  Information 


•  Jeremy  Dusina 

-  ATEC-AEC-AVED 

-  Comm:  (703)  681-2729 

-  DSN:  761-2729 

-  BB:  (571)309-9873 

-  Jeremv.Dusina@us.armv.mil 

•  Juan  Carlos  Sanchez 

-  ATEC-AEC-AVED 

-  Comm:  (703)  681-2733 

-  DSN:  761-2733 

-  JuanCarlos.Sanchez@us.armv.mil 
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NORTHROP  GRUMMAN 


Defining  the  future 

Unmanned 
Systems: 
What's  New, 
What's  Not 
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\ 

Gene  Fraser 

Vice  President  and  Deputy,  Integrated  Systems  Western  Region 
Unmanned  Systems  Lead  and  Site  Manager 
Northrop  Grumman  Corporation 


t  Sales 


Northrop  Grumman  2007  Sales 


Northrop  Grumman 
Corporation 

Total  Sales  2007:  $32B* * 


OPERATING  SECTORS 


MISSION  INFORMATION  TECHNICAL  ELECTRONIC 

SYSTEMS  TECHNOLOGY  SERVICES  SYSTEMS 


$4.5B 

Enterprise  Systems 
and  Security 

IT/Network 

Outsourcing 

Defense  & 
Intelligence 

Federal,  State  /  Local 
&  Commercial 


$2. IB 

Systems  Support 

Training  and 
Simulations 

Life  Cycle 
Optimization 


$6.7B 

Radar  Systems 
C4ISR 

Electronic  Warfare 

Navigation  & 
Guidance 

Military  Space 
Homeland  Defense 


Command,  Control 
and  Intelligence 

Land  Forces  / 
Digitized  Battlefield 

ICBM  Systems 
Management 


Missile  Defense 
BMC3 


$6.1B 


Information  Warfare 


ISR,  Homeland 
Security  &  Health 


Government  Systems 


INTEGRATED 

SYSTEMS 


SPACE 

TECHNOLOGY 


SHIPBUILDING 


$5.1B 

Large  Scale  Systems 
Integration 

C4ISR 

Unmanned  Systems 

Airborne  Ground 
Surveillance  /  C2 

Naval  BMC2 

Global  /  Theater 
Strike  Systems 

Electronic  Combat 
Operations 


$3.1B 

ISR  Satellite  Systems 

Missile  Defense 
Satellite  Systems 

MILSATCOM  Systems 

Environmental  & 
Space  Science 
Satellite  Systems 

Software  Defined 
Radios 

Directed  Energy 
Systems 

Strategic  Space 
Systems 


$5.8B 

Naval  Systems  Integrator 
Nuclear  Aircraft  Carriers 
Nuclear  Submarines 
Surface  Combatants 

Expeditionary  Warfare 
Ships 

Auxiliary  Ships 
Fleet  Maintenance 

Marine  Composite 
Technology 

Coast  Guard  Cutters 
Commercial  Ships 


Information  &  Services 


Electronics 


Aerospace 


Shipbuilding 


A  Portfolio  Positioned  for  the  Future  Northrop  Grumman 

*  Includes  Intersegment  Eliminations 

f  Realigned  Results  Approved  for  Public  Release  -  Corp.  2008 


Capability  Developments  for  the  Future  are 
Built  Upon  a  Legacy  of  Unmanned  Aerial  Systems 
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Unmanned  Systems 
I  ntegrated  Capabilities  and  Support 
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J  ■  Global  Hawk 
Flight  Test 


UMS  Support 
Center 


Global  Hawk 
CONUS  Flight  Ops 


Targets  Operations 


El  Segundo 
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VTUAV 
Flight  Test 

GHMD  Flight 
Test 


Global  Hawk  Engineering 
X-47B  UCAS  Engineering 
Fire  Scout  Engineering 
Targets  Development 
Rapid  Prototyping 


X-47B  UCAS 
Engineering 


Global  Hawk  Production 
Targets  Production 
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Unmanned  Systems  are  Required  Globally 
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Key  Capability  Developments  for  the  Future 

Creating  an  Autonomous  Environment 


Aerial  Refueling 


Virtual  Warfare 
Environment 


Cooperative  Control 
and  Mission  Management 


Command  and 
Control  Flexibility 


Product  Development  I  nvestments 
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End  to  End  System  I  ntegration... 


...Key  to  Success  to  Date 
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Unmanned  Systems  Current  Contracts 


Global  Hawk  Customer:  US  Air  Force 

A  combat  proven  high  altitude,  long 
endurance  UAS  with  extraordinary  ISR 
capabilities.  Can  provide  near  real  time 
high  resolution  imagery  of  large 
geographical  areas  all  day/all  night  in  all 
types  of  weather. 


GHMD 


Customer:  US  Navy 


Maritime  version  of  the  high  altitude,  long 
endurance  Global  Hawk  UAS  with 
application  for  broad  area  maritime 
surveillance  mission. 


Euro  Hawk 


Customer:  German  MoD 


First  International  version  of  high  altitude, 
long  endurance  Global  Hawk  UAS  with 
German  ELINT/COMINT  sensor  on  board. 


VTUAV  Fire  Scout  Customer:  US  Navy 

Navy  multi-mission  system.  Operates 
autonomously  from  any  aviation-capable 
warship,  or  unprepared  landing  zones  in 
battle  area. 


FCS  Class  IV  UAV  Customer:  US  Army 

Army  Future  Combat  Systems  multi¬ 
mission  Class  IV  UAV  for  the  Brigade 
Combat  Team  can  provide  RSTA, 
countermine,  chemical  detection,  TUGS 
deployment,  and  logistics  re-supply. 


UCAS-D 


Customer:  US  Navy 


Navy  UCAS  Carrier  Demonstration  will 
show  that  an  operational  class,  tailless, 
LO-representative  air  vehicle  can  operate 
from  a  carrier  at  sea. 


Hunter 


Customer:  US  Army 


Army  Reconnaissance,  Surveillance  and 
Target  Acquisition  (RSTA)  system 
currently  in  service.  Combat  proven  in 
multiple  theaters  and  deployed  in  Iraq. 


J argets  Customer:  Multiple 

1  Low  cost  UAS  for  simulation  of  threat 
aircraft  and  missiles  for  combat  training 
and  weapons  evaluations. 


Model  324 


Customer:  Egyptian  Air  Force 


Autonomous,  long  range  reconnaissance 
UAS  providing  EO  and  IR  imagery. 
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Defining  the  future1 


Dynamic  Program 

Upda  tes 


Global  Hawk 
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Combat  proven  high 
altitude,  long 
endurance  UAS  with 
extraordinary 
Intelligence, 
Surveillance, 
econnaissance  (ISR) 
capabilities. 

rovides  near  real  time 
high  resolution 
imagery  of  large 
geographical  areas  all 
day  /  all  night  in  all 
types  of  weather. 
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Global  Hawk  Maritime  Demonstration 


Maritime  version  of 
the  high  altitude, 
long  endurance 
Global  Hawk  UAS 
with  application  to 
the  broad  area 
maritime  surveillance 
Concept  of 
Operations 
(CONOPS) 
development. 
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Euro  Hawk 
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Contracted  in  January 
2007  with  the  German 
Ministry  of  Defence, 
Euro  Hawk  represents 
the  first  international 
development  program 
for  Global  Hawk. 
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Ship- Based  Fire  Scout  VTUAV  System 


VTUAV  Control  Station 
Integrated  Into  the 
Ship's  Combat 
Information  Center 


X-47B  UCAS 


Carrier  Demonstration  (UCAS-D)  program  - 


demonstrating  the  ability  of  an  unmanned,  low 
observable  platform  to  conduct  operations  from  a 
carrier  at  sea  to  include:  launch,  recovery,  deck 
handling  and  full  integration  into  Carrier  Air  Traffic 

Control  Center. 
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New  Unmanned  Systems  Pursuits 


Navy  UCAS  Customer:  US  Navy 

Completion  of  the  UCAS-D  program  in 
2013  is  the  first  step  toward  an 
operational  Navy  UCAS  strike  /  ISR 
platform. 


Customer:  Multiple 


Fire  Scout 
International 

Multi-mission  system  which  has 
been  source  of  many  international 
requests  to  USN.  High  degree  of 
interest  in  Europe  and  Asia. 


AGS  Customer:  NATO 

NATO  System  consisting  of  a 
combination  of  manned  and  unmanned 
systems  to  provide  ISR  capabilities. 
Unmanned  component  is  a  Global  Hawk 
UAS. 

USCG  FS  Customer:  USCG 

Potential  Fire  Scout  application  for 
U.S.  Coast  Guard.  Autonomous 
system  that  can  land  on  any  air 
capable  ship. 


BAMS  Customer:  US  Navy 

Maritime,  high  altitude,  long  endurance 
Intelligence  Surveillance 
Reconnaissance  and  Targeting  (ISR&T) 
system  based  on  combat-proven 
Global  Hawk. 

Customer:  USMC 

Potential  Fire  Scout  application  for 
Marines.  Autonomous  system  that 
can  land  in  battlefield  areas  and 
act  as  communications  relay. 

Targets  International  customer:  Multiple 

Low  cost  UAS  for  simulation  of 
threat  aircraft/missiles  for  combat 
training  and  weapons  evaluations. 
Currently  sold  to  various  nations. 

ACIS  Projects  Customer:  Multiple 

The  Advanced  Capabilities  and  Integrated 
Solutions  Projects  Team  leads  the  way  for 
the  developmental  and  implementation  of 
new  technologies  that  will  enhance  the 
capabilities  of  unmanned  systems. 
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Continuing  to  Support  the  Warfighter... 
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Presentation  Overview 


ft  ASTM  F38  Mission  &  Vision 
ft  How  we  can  help  you 

•  Published  &  Draft  Standards 

ft  Background  on  ASTM  International  § 
*  Some  Specifics  about  F38 
ft  Questions 
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Committee  F38 


Vision :  Provide  routine,  safe  UAS  operations  in  civil  airspace 
through  standardization. 


Mission :  Produce  practical,  consensus  standards  that  facilitate  UAS 
operations  at  an  acceptable  level  of  safety  for  use  by  industry, 
academia,  government  organizations  and  regulatory  authorities. 

* 

«  Guiding,  Principle:  Practical. standards  are  a  cost  effective  means  of 
promoting  commercial  success,  and  that  consensus  processes 
protect  the  balance  of  interests  among  stakeholders. 


A  Spectrum  of  Standards 


Voluntary  Mandatory 

Unregulated  Industry  Industry 

Standards  Standards 


Heavily 

Regulated 


Kites 

Balloons 

Models 


Large  Aircraft 
Airlines 
Pilots 
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F38  Subcommittee  Structure 


What  do  you  need  to  fly? 
...A  System  Safety  Case 

Airframe  certification 


Operations  protocols  & 
component  performance 


F38.01  Airworthiness  Standards 


m 


•  Safe  design,  construction,  test, 
modification,  &  inspection  of  the 
individual  component,  aircraft,  or 
system;  hardware  oriented 

F38.02  Operations  Standards 


Safe  employment  of  the  system  within 
the  aviation  environment  among  other 
aircraft  &  systems;  procedure/ 


performance  oriented 


Crew  training  &  human  factors 
consideration 


F38.03  Pilot  &  Maintenance  Qualifications 

•  Safe  practices  by  the  individuals 
responsible  for  employing  the 
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system;  crew  oriented 


F38.01:  Subcommittee  on  Airworthiness 


What  do  you  need  to  fly? 

•  System  certification 

•  Operations  protocols  and  component  performance 

•  Crew  training  and  human  factors  consideration 

You  would  need 

•  Reliability  and  Airworthiness  Standards 

-  Aircraft,  Control  Station,  Datalink 

•  Support  Equipment  Standards 

-  Launch  &  recovery  equipment 

-  Starters,  power  supplies,  fueling  /  de-fueling,  others 
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F38.02  Subcommittee  on  Flight  Operations 


What  do  you  need  to  fly? 

*  Airframe  certification 

*  Operations  protocols  and  component  performance 

*  Crew  training  and  human  factors  consideration 

You  would  need 


•  Standardized  flight  procedures 

*  Standardized  maintenance  procedures 

•  Safe  separation  from  other  airspace  users 

*  Others,  of  course 
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F38.03  Subcommittee  on  Personnel 


What  do  you  need  to  fly? 

•  Airframe  certification 

•  Operations  protocols  and  component  performance 

•  Crew  training  and  human  factors  consideration 

You  would  need 

•  Pilot  certification  system 

-  Category  and  type,  ratings,  limitations 

•  Criteria  to  certify  aircrewmen 


-  Eligibility,  Knowledge,  Experience,  Test  Standards 

*  Criteria  to  certify  maintainers 

•  Others,  of  course 


How  We  Can  Help  You:  Published  Standards 


F2395-07 

Standard  Terminology  for  Unmanned  Aircraft  Systems 

F2501-06 

Standard  Practices  for  UAS  Airworthiness 

F2585-06 

Design  &  Performance  of  Pneumatic-Hydraulic  Launch  System 

F2500-07 

Standard  Practice  for  UAS  Visual  Range  Flight  Operations 

F2584-06 

Standard  Practice  for  Maintenance  &  Manuals  for  Light  UAS 

F2612-07 

Standard  Practice  for  Design  and  Manufacture  of  Turbine  Engines  for 
Unmanned  Aircraft  Systems 

F2512-07 

Standard  Practice  for  Quality  Assurance  in  the  Manufacture  of  Light  Airplane 
Unmanned  Aircraft  Systems 

F2667-07 

Standard  Practice  for  Design  and  Manufacture  of  Reciprocating 

Compression  Ignition  Engines  for  Unmanned  Aircraft  Systems 

F2635-07 

Standard  Classification  for  Unmanned  Aircraft  Pilot  Certification 

F2636-08 

Commercial  UAS  Pilot  Practical  Test  Standards 
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How  We  Can  Help  You:  Items  In  Work 


WK11425 

Private  UAS  Pilot  Practical  Test  Standards  (Dave  Gibbs) 

WK13935 

Standard  Guide  for  Mini-UAS  Airworthiness  (Jason  Stiffey) 

WK13989 

Standard  Practice  for  Mini-UAS  Visual  Range  Operations  (Dave  Grilley) 

WK12989 

Standard  Practice  for  Mini-UAS  Operators  (Dave  Grilley) 

WK8962 

Standard  Practice  for  Remote  Control  Pilots  Operating  within  Visual  Range 
(Dave  Grilley) 

WK13686 

Suggested  Procedures  Guide  for  Applying  for  UAS  Special  Issuance  and 

Type  Certificates  (Dr.  Gerry  Marsters,  former  Transport  Canada  Regulator) 

WK15881 

Specification  for  Design  and  Performance  of  UAS  Recovery  Systems 

Leveraging  Community  Expertise 
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Community  Value 


Applying  These  Standards 

•  Package:  Mini-UAS  in  Visual  Range 

-  Airworthiness  (WK13935) 

-  Operations  (WK13989) 

-  Pilots  (WK12989) 

•  Creates  a  Safety  Case 

-  For  Regulators 

-  For  Insurers 

Buyer  /  User  Adoption 


•  Simplifies  procurement  process 

*  Enables  interoperability 
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ASTM’s  Standards  Development 


A  Proven  and  Practical  System  that  is  Driven  by  Direct-Stakeholder 
Participation,  for  Developing  Voluntary,  Consensus  Standards  for  Materials, 
Products,  Systems  &  Services  World-Wide. 


A  Portfolio  of  Approximately  2,000  Standards  Used  Internationally;  3,500 
are  the  Basis  of  National  Standards  and  Regulation  in  76  Countries. 

Always  Reflect  Current  Technology  as  they  are  Continually  Revised. 

T 

Over  31,000  Members  rorn  130  Countries  Participate  on  ASTM  International 
Committees;  users  from  175  countries. 

Standards  Development  Process  complies  with  WTO’s  TBT  Requirements. 

*  No  Project  Costs. 
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140  Technical  Committees 


A  FEW  OTHER  EXAMPLES 


A01  on  Steel,  Stainless  Steel,  &  Related  Alloys 
tii  B07  on  Light  Metals  &  Alloys 

D02  on  Petroleum  Products  &  Lubricants 
D20  on  Plastics 

E34  on  Occupational  Health  &  Safety 
tii  E54  on  Homeland  Security  Applications 

•  E54.08  on  Operational  Equipment  /  Urban  Search  &  Rescue  Robots 

F04  on  Medical  Devices 
tii  F37  on  Light  Sport  Aircraft 

tii  F39  on  General  &  Utility  Category  Aircraft  Wiring  Systems 
tii  F41  on  Unmanned  Undersea  Vehicle  Systems 


INTERNATIONAL 
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What  is  a  “Consensus  Standards  Body” 


Attributes 

•  Openness  with  a  “balance”  of  interest 

•  Formal  processes  including  appeals 

•  Consensus  (vice  unanimity) 

-  Must  include  a  method  for  resolving  negatives 

What  is  not 

•  Company  standards 

•  Government  standards 

•  Standards  mandated  by  law 


•  Market  driven  “de  facto”  standards 
-  Examples:  VHS,  MS  Windows 


Defining  F38’s  “Balance” 


Member  Type 

•  Individual:  $75  annual  dues 

•  Corporation:  $400  annual  dues 

•  Temporary:  Courtesy  trial  membership 
Classification 

•  Producer:  Seller  of  products  and  services 

•  User:  Buyer  of  products  and  services 

•  General:  Other  interested  parties 


w  Voting  Status 
•  Tracked  by: 


-  Type 


-  Classification 


INTERNATIONAL 


-  Interest  (i.e.,  company  or  organization 
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215  Members 


•  18  Countries  /  4  Continents 


Australia 

1 

Bahamas 

1 

Canada 

5 

Chile 

1 

Finland 

1 

France 

1 

Germany 

6 

Israel 

2 

Japan 

6 
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Republic  of  Korea  1 


New  Zealand  1 


Singapore 


Slovenia  1 


Spain  1 


Sweden 


Taiwan  3 


United  Kingdom  2 


United  States  177 
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F38  Total  Membership 
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Proportionally  balanced 
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99 

45 

71 

1  1 

1 
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QUESTIONS? 


Daniel  A.  Schultz 

ASTM  International 
Manager,  Standards  Development 

1+610-832-9716 

dschultz@astm.org 

www.astm.org 

100  Barr  Harbor  Drive 
W.  Conshohocken,  PA  19428-2959 
USA 


Jim  Jewell 

UAV  Markets  pace 

F38  Membership  Secretary 

+1  (516)  558-7981 

iames@uavm.com 


PO  Box  330 

Oyster  Bay,  NY  11771 

USA 
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Committee  Chairman 
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Co-Chairman 

27  February  2008 


NATIONAL  DEFENSE 1N01  'SIHIAL  ASSOC 1ATI0N 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


wv^vm  ICOTE  Organizing 

Principles 


The  objectives  of  the  ICOTE  are  to: 

•Provide  a  forum  for  discussion  and  exchange 
of  T&E  views  from  Defense  and  the 
Industrial  Base  in  a  non-attribution 
environment 

•Gain  feedback  from  senior  industry 
representatives  and  OSD  leadership 

•Discuss  OSD  and  service  policies  which 
affect  relationships  with  industrial  suppliers 

•Discuss  emerging  issues  /  policies  in 
government  and  industry  which  affect  the 
readiness  and  capabilities  of  U.S.  defense 
system  producers 

•Cooperate  on  various  projects  of  mutual 
benefit  to  the  ICOTE  participants 


STKKNCTH  TUR  N  Gil  IMDiSTRV  A  TECHMOIOGV 


ICOTE 

Current  Membership 


CURRENT  NDIA  T&E 
ICOTE  PARTICIPANTS 

NDIA  PARTICIPANTS 

Lt  Gen  Larry  Farrell,  USAF  (Ret)  Pres  &  CEO,  NDIA 
Sam  Campagna;  Director  of  Operations 
James  O’Bryon:  Chair  T&E  Division 

GOVERNMENT  MEMBERS 

Dr.  Charles  McQueary  DOT&E 
David  Duma  DOT&E 
Dr.  Ernie  Seglie  DOT&E 
RDML  Bill  McCarthy,  USN  ret.  DOT&E 
Michele  Williams  NGA 
Brian  Simmons  AEC 
Mike  Crisp  DOT&E 
Maj  Gen  Roger  Nadeau  ATEC/CG 
Maj  Gen  Steve  Sargeant  AFOTEC  CC 
RDML  Stephen  Voetsch  OPTEVFOR 
B/G  David  Eichhorn  Edwards  /CG 
Colonel  Mike  Bohn  MCOTEA 
Dr.  Steve  Hutchison  DISA 
Jim  Streilein  ATEC 
Steve  Whitehead  OPTEVFOR 
Juan  Vitail  PEO  CBD 


STttK\i;TllTllft>lfJI  IMDiSTRV  A  TECHMOIOGV 


ICOTE 

Current  Membership 


CURRENT  NDIA  T&E 

ICOTE  PARTICIPANTS 

INDUSTRY  MEMBERS 

Oscar  Arroyo  Raytheon 
Brian  See  ATK 
Regis  Luther  BAE 
Steve  Zink  Oshkosh 
Jim  Ruma  General  Dynamics  Land  Sys 
Martin  Peryea  Bell  Hellicopter 
Parker  Horner  EWA 
Bill  Keegan  SAIC 
Steve  Kimmel  Alion 
Bill  Shane  Boeing 
Jim  Vosper  Boeing 
Doug  Pearson  Lockheed  Martin 
Joe  Sweeney  Lockheed  Martin 
Jon  Neasham  Cubic 
Yank  Rutherford  Northrop  Grumman 
Gene  Fraser  Northrop  Grumman 
Tom  Quinn  BAH 
Larry  Graviss  Eagle  Engineering 
Jim  O’Bryon,  O’Bryon  Group 


iHiriim 

STRENGTH  TIIIKKXJII  EMU  STAY  &  TECHNDUKY 

ICOTE 

•  MEETINGS 

The  ICOTE  will  meet  at  the  call  of  the 
Chairman  and  Co  Chairman  at  sites  and 
times  convenient  to  the  members. 

Minutes  of  the  ICOTE  will  be  provided  to 
all  members  in  the  form  of  Agendas  and 
action  items  as  determined  by  the  Director 
ofOT&E. 

Topics  of  interest  for  consideration  by  the 
ICOTE  will  be  solicited  from  industry  and 
government  members. 


iWiarii'%  ICOTE  Themes  for 

STRENGTH  TUROttll  [MX5TRV  &  TCCHHOIOG* 

The  past  year 


Themes  for  2007  /2008 

Temp  Process  Change  -  DISA 
I A  Test  Policy,  CND  testing  10  range 
Mil  Std  785  B  and  using  industry  std. 
Special  Subjects 

Sustainment  and  Evolutiona  ry  AQ 
MRAP  RFP 

FCS  Sustainment  Program 
MDA  Test  Methodology 
JIEDDO  approach  to  T&E 
Threat  Based  Methodologies  @  MS 
DISA 

USAF  Test  Command 
APG  Frag  Kit  6  /  MRAP  testing 
M&S  CVN21 

ICOTE  Sub-Committee  on  Space 
Reports 
•  DSB  Report 


OF  UKVs 


Hubert  Hecht,  PhJDl 
'SoHiiR  Incorporated 
Culver  City,  California 


NO  HUMAN  PILOT 


*  SAVES  WEIGHT 

•  SIMPLIFIES  DESIGN 

BlNC^pSpS-LOSS 

-ACCE^NGE 

gEXTENDS  FLIGHT 
ENVELOPE' 


J  ALL  HANDLING  OF 
ANOMALOUS 
CONDITIONS  MUST 
3E  PROGRAMMED 
AMD  TESTED  IN 
ADVANCE 


EXCEPTION  HANDLING 


REQUIREMENTS  BASED  STRUCTURAL 


gpLL  ST/fTIP,  REQUIRE* 
MENTS  HAVE  BEEN 
IM^EMENTEO  ^  - 

fi  ExFcdloN  PRODLT- 

CEsSESIRllf^/  . 
REslifs  - 


•  TRAVERSAL  OF 
IMPLEMENTED 
PATHS  PRODU¬ 
CES  NO 

'UNDESIRABLE 
RES  UR'S 


NEITHER  APPROACH  ASSURES  ADEQUACY  OF 

EXCEPTION  HANDLING 


HANDLING 


•  VERY  LITTLE  LITERATURE 

.  EXCEPT  FOR  LANGUAGE  CONSTRUCTS 

♦Jn6  GUIDANCE  TOR  SYSTEM  LEVEL'  ' 
REQUIREMENTS  FORMULATION 

•  MOSj  SOFTWARWFA'|LUr|s  IN  WELL- 
TESTED  SYSTEM  ARE  DUE  TO  FAULTY 
EXC^PlSoi HANDLING 


critotoity 


SRJgglSHgHLMMpNICS  SOFTWARE 


Hecht,  H.  and  P.  Crane,  “Rare  Conditions  and  their  Effect  on  Software 
Failures”,  Proc.  of  the  1994  Annual  Reliability  and  Maintainability 
Symposium ”,  January  1994,  pp.  334  -  337. 


MOJRE  EXCEPTION  HPSIDLPG 

FAILURES 


0^ 

ALL  FAILURES  GLOBAL  FAILURES 


Kanoun,  K.  and  T.  Sabourin,  “Software  Dependability  of  a  Telephone 
Switching  System”,  Digest  of  Papers,  FTCS-17,  Pittsburgh  PA,  July  1987, 
pp.  236-241 


RELEVANT  QUOTES 


“The  main  line  software  code  usually  does  its  job.  Breakdowns  typically 
occur  when  jhe  software  exception  code  does  notproperly  handle 
IbndMdl  mpufor  environmental conditions  of  when  an  interface 
does  not  respond  in  the  anticipated  or  desired  manner.” 


C.  K.^ansen^Tfte  Status  of  Reliability  Engineering  Technology  2001 ,  Newsletter  of  the  lEElS 
Reliability  Society,  January  2001 


^herefd^re  the  identification  and  handling  of  the  exceptional  situations 
thatfpight  occur  is  often^just  as  (un)reli£ble  as  human  intuition.” 

Flaviu  Cristian  “Exception  Handling  and  Tolerance  of  Software  Faults”  in  Software  Fault  Tolerance, 
Michael  R.  Lyu,  ed.<Wiley,  New  York,  1995 


SPECIFYING  EXCEPTION 

hIndliiJp  is  PfficI  LIT 

•XXClF^bRcOSDjilpNS  ARJSE  FROM 

SWERAL  lIve'ls 


SPECIFYING  EXCEPTION 

hSndliiJp  is  PfficI  LIT 

MxciPmoWooBp'mpHs  arjse  from 
sS/Jral  lIve'ls 

•  ixclp®o|coib®ldNS-KRE'MORE 

DIFFICULT  TO  UNDERSTAND  THAN 
MAIN  LtNCRTIqU I REMEHtS 


SPECIFYING  EXCEPTION 

hIndliiJp  is  PfficI  LIT 

-  EXCEPTION  CONDITIONS  ARISE 
pkOM  SEVERAL  LEVELS 

•  EXClpfFolcoflblfldNS-KRE'MORE' 

DIFFICULT  TO  UNDERSTAND  THAN 
MAIN  LINE  REQUIREMENTS 

•  EXCEP|WOpS  OCCUR  INFREQUENTLY 
BUT  REQUIRE  DISPROPORTIONATE 

■SfforbP^  X' 


SOURCES  OF  EXCEPTIONS 

OPERATIONAL  REQUIREMENTS 

LOSS  OF  PROPULSION,  ELECTRIC  POWER,  COMMUNICATION,  THERMAL 
CONTROL 

IMPLEMENTATION  DETAIL 

CALIBRATION  ANOMALIES,  ACTUATOR  STATES,  SENSOR  INPUT 

COMPUTING  ENVIRONMENT 

HARDWARE  FAILURES,  MEMORY  ERRORS,  EXECUTIVE,  MIDDLEWARE 

MONITORING  AND  SELF-TEST 

OVER-TEMPERATURE  SENSORS,  SYSTEM  PERFORMANCE  TEST 

APPLICATION  SOFTWARE 

ASSERTIONS,  VIOLATION  OF  TIMING  CONSTRAINTS,  MODE  CHANGES 


WHO  IS  RESPONSIBLE? 


OPERATIONAL  REQUIREMEN 
MRLEMENTATION  DETAILS 


>  SYSTEM 
ENGINEERING 


SPECIALIST 


G  ENVIRONMENT 


VEHICLE 


MONITORING  AND  SELF-TEST  health mgmt 


APPLICATION  S 


r\  I  v  vr-vi 
SOFTWARE 

ENGINEERING 


GENERATION 


•  OBJECTIVE 

.  EXCEPTION  CONDITION  AndKctTON 


*  ALGORITHM 

^^QUANTITATIVE  CONDITIONDESCRIPTION 
^TIIVflNGKiTD^RESPONSIBILITY  FOR 

^PCction 


f\SS\G  N  ' 

"^SPlcIfY  SoffWARE  IMPLEMENTATION 

of!algorithm 


DOES  IT  ADD  UP? 


OBJECTIVE  ALGORITHM  ASSIGNM'T 


OPERATIONAL  REQM'TS 


OBJECTIVE  ALGORITHM  ASSIGNM'T 


IMPLEMENTATION 


OBJECTIVE  ALGORITHM  ASSIGNM'T 


COMPUTING  ENV. 


MONIT.  &  SELF-TEST 


OBJECTIVE  ALGORITHM  ASSIGNM'T 


APPLICATION  SOFTW. 


OBJECTIVE  ALGORITHM  ASSIGNM'T 


CONCEPT  SYST.  REQ'MTS  SOFTW. REQ'MTS  SOFTW. DESIGN 


CODING 


•SSHARiraQ'EXISl  NG  PRACTICES 
•  SHARING  EXPERIENCE 
•j  CRYING  AND  SHArR|NG'fpOLS 

•JintSres?  GROUP  - 

■  SpVHDARDS  WORK*G  GROUP 
<  ^ECOMMEWDED  PRACTICE 


HERBERT  HECHT 

herb@sohar.com 

310.338.0990X110 


nsara 


New  Mexico  State  University/ 

2  Physical  Science  Laboratory 
Technical  Anatysis  and  Applications  Center 
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v 
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I 


RDT&E  to  Advance  UAS  Access  to  the 

-• 

National  Airspace  System 

.  —  — ~  — ■ " 

j-  Steve  Hottman 

I  _ 

February  28,  2008 

24th  Annual  National  Test  &  Evaluation  Conference" 
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Physical  Science 
Laboratory 


JZXftC 


NMSU/Physical  Science  Laboratory  Overview 


•  Established  in  1946  to  support 
missile  testing  of  V-2/Aerobee 
rocket  testing  at  WSPG 


•  Multi-disciplined,  aerospace-  and 
defense-oriented  scientific  and 
technical  organization 

•  A  TOP  SECRET  cleared  facility 


CVJ  Ai'vs  <$■ 

Ajj,  ^  ^ 

A  VS.e5>.  ^  . 

(l\  ^  ^ 

C£ ") 

«ss»  jO  Os2r&^-  ■ 

<d\ 

Jlj/y^J-  A  .  A 

.  Tr-r. 
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Physical  Science 
Laboratory 
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UAS  Demonstrations  and  T&E 
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Physical  Science 
Laboratory 


WSMR 


Las  Cruces 


T&E 


Denied  GPS 
Environment 


Asymmetric  Threat 


IED  Demo 


Playas 


Orbiter 


Physical  Science 
Laboratory 


JZXftC 


Detect,  Sense,  and  Avoid 


i: 


□  200] 


Avoidance 


75  Left 


CPJJ 


Basler  Cameras 
Octec  Tracker  Cards 

75  Right 


Ttopir  Geneva  FlightTEK 
Interface  Autopilot 
Pane 


"•  Auto-pilot  Servos 


Sentinel  and  SAVDS 


ERAST 


ATDSS-III 
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Alaska 


NM 
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UAS  Research 


NM 


Physical  Science 
Laboratory 
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Air  Traffic  Control  Research 
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Physical  Science 
Laboratory 
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Approach 


Field  evaluation  of  UAS  symbology  at  FAA  Air  Traffic  Control 
locations 

•  Aircraft  call  sign 
Participants 

•  Seattle,  Albuquerque,  Fort  Worth,  Denver,  New  York  En 
Route  Centers  and  New  Orleans,  Denver,  High  Desert 
(Edwards  AFB),  Tucson,  Albuquerque  TRACONs 

•  53  ARTCC  controllers,  46  TRACON  controllers 


NM 


Physical  Science 
Laboratory 


me 


Potential  UAS  Symbology 


•  Aircraft  call  signs  are  used  by  air  traffic  controllers  to  identify 
individual  aircraft: 

-  UAV173  ^ 

-  UM9417 

-  UIN237  (uses  aircraft  registration  number) 

-  UN4237  (uses  aircraft  registration  number) 

•  The  data  block  (right)  appears  on  the  controller’s  radar  scope.  UAV 173 

Flight  progress  strips  (below)  are  printed  on  pieces  of  paper.  1 80 C 

426  223 


UM9417 


T/PRED/G 
T454  G499 
89 

425  04/1 


sso 

360 

017 

0012 


00 

31 

EWM 

370 


SNA./.DR  EWM  J4  INK 
JEN5  DFW 


o  UAV  TYPE  2 
PREDATOR 


6722 
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UAS  Operator  Requirements 


>  Landing  task  description 
3  miles  from  airport 
A-  Lined  up  on  centerline 
>>  500’  AGL 
4-  Substantial  crosswind 

Runway  -  200’  width;  7000’  long 
4-  Acceptable  landing  parameters 
—  Runway  location 
—  Heading 
—  Vertical  velocity 


Results 

•  Data  included  as  part  of  book  chapter 

“Required  Attributes  and  Skills  of  UAV 
Operators”  for  the  future  Human  Factors  of 
Remotely  Piloted  Vehicles  volume  of  the 
Advances  in  Human  Performance  and  Cognitive 
Engineering  Research  series. 


Physical  Science 
Laboratory 
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Handling  Qualities 


•  Developing  an  assessment  scale  for  UAS  handling 
qualities.  Derived  from  the  Cooper-Harper  aircraft 
handling  qualities  scale 

•  The  UAS  handling  qualities  scale  will  be 
multidimensional  (unlike  Cooper-Harper),  non- 
intrusive,  and  will  not  compromise  flight  safety. 

•  The  development  of  the  UAS  handling  qualities 
scale  will  involve  two  empirical 

phases:  dimension  identification  and 
validation.  After  scale  construction,  both  content 
validity  and  inter-rater  reliability  will  be 
empirically  evaluated. 


NM 
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Trust  in  Automation 


•  Conducted  an  experiment  that 
found  time  pressure  increased 
trust  and  compliance  in 
automation.  This  is  beneficial  to 
overall  human-automation 
performance  when  the  automation 
is  highly  reliable. 

•  Second  experiment  indicated  that 
this  increase  in  trust  carries  over 
to  second  session  even  when  time 
pressure  is  removed. 

•  Third  experiment  indicated  that 
time  pressure  is  only  effective 
when  task  is  difficult  and 
participants  feel  that  the 
automation  is  doing  a  good  job. 

•  Fourth  experiment  reveals  a 

function  by  which  the  more  time 
given  to  complete  the  task,  the 
less  compliance  participants  have 
in  the  automation.  _ 


Accuracy  as  a  function  of  time  allowed 


NM 
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UAS  Flight  Test  Center 
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UAS  Regulatory  Status 

•  No  regulations  exist  for  UAS;  only  guidance 

•  No  empirical  data  exists  to  help  drive  regulatory 
development 

•  Access  to  airspace 

•  Civil  -  Experimental  Airworthiness  Certificate 


REGISTRATION  NOT  TRANSFERABLE 


LSHITEB  STATES  OF  AMCAiCA 

HEPA*TM£NT  OF  TRANSPOBTATlON  -  FE-OERAL  AVWTIDN  APMIHISTHATIQH 
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Need  for  UAS  Flight  Test  Center 

•  CO  A  process  used  for  public  aircraft  operators 

•  Experimental  airworthiness  certificates  required  for 
commercial  operators 

-  Data  must  be  generated  to  substantial  airworthiness 

•  To  generate  data,  you  must  be  able  to  fly 

•  To  fly  in  the  NAS,  you  must  have  an  experimental  airworthiness 
certificate 

•  To  obtain  an  airworthiness  certificate,  flight  data  are  required 

•  No  authorized  flight  areas  exist  for  UAS 

•  FAA  needs  data  for  development  of  regulations 


NM 


Physical  Science 
Laboratory 
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Why  A  Test  Center? 

•  UAS  are  Different 

-  Manned  Aircraft  under  testing  and 
development  can  comply  with  14 
CFR  Part  91 

-  Private  Industry  needs  a  place  to  do 
basic  Research  and  Development 

•  Many  are  not  ready  for  the 
FAA  Experimental 
Certification  process 

•  Not  just  for 

aircraft . payloads 

-  UAS  are  still  very  immature 


Why  NMSU? 

►  Experience  with  UAS 

-  Foundational  SOP’s  in  place  and  exercised 

-  Ability  to  collect  and  process  significant  data 

-  Solid  and  credible  safety  record 

•  Over  8  years  operational  experience 

-  Experienced  UAS  personnel 
•  Location  - 

-  “It’s  not  the  end  of  the  world,  but  you  can  see 

it  from  there . ” 

-  Very  sparsely  populated 

-  Low  density  Air  Traffic 

-  Climate  is  favorable 


How? 


Establishing  a  Cooperative  Research  and  Development  Agreement  (CRD A) 

-  Outlines  the  risk  management  process  similar  to  that  being  applied  by  FAA  today 
FAA  will  require  data  on  a  routine  basis 


NMSU  is  a  Public  Organization  and  thus  qualifies  for  a  COA 
-  All  testing/R&D  will  be  conducted  as  directed  by  FAA  under  the  COA 


Provides  for  a  controlled  testing  environment  while  minimizing  impact  to  other  NAS  users  as  well  as 
people/property  on  the  ground 


NM 


Physical  Science 
Laboratory 
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QuickTime™  and  a 
TIFF  (LZW)  decompressor 
are  needed  to  see  this  picture 


NM 

STATE 


Physical  Science 
Laboratory 
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Southwest  New  Mexico  Airspace 


•  Largest  DoD- 
controlled  air  and 
ground  space  in  the 
U.S. 

•  Slightly  smaller  than 
Connecticut  -  7,105 
sq  mi 

•  USAF  air  traffic 
control  from  “surface 
to  space” 

•  NMSU/PSL  COA 
>12,000  sq  mi 


NM 


Physical  Science 
Laboratory 
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Sweden/Arctic 


•  Currently  performing 
Arctic  airspace  study  with 
FAA  UAPO 

•  Arctic  overflights 
proposed  since  2003 

•  CY05  USA/Sweden 
Space  Exploration 
Agreement  and 
International  Polar  Year 
with  Swedish  Space 
Corporation 

•  High-altitude  balloon 
experience 


Physical  Science 
Laboratory 
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Summary 

•  Routine  access  to  the  NAS  is  not  yet 
available 

•  The  regulatory  body  for  UAS 
requires  development 

•  Significant  RDT&E  is  required 
before  realistic  “file  and  fly”  in  the 
NAS  exists 

•  Formal  studies  with  resulting 
empirical  data  will  assist  with  FAA 

certification  issues 

UAS  Flight  Test  Center  provides 
NAS  access  and  a  T&E 
environment 

PSS?””  21  me 


Joint  Mission  Environment 
Test  Capability  (JMETC) 


Briefing  for: 

NDIA  National  Test  and  Evaluation  Conference 

February  27,  2008 


Agenda 

•  Interoperability/NR  KPP  Test  Requirement 

•  Program  Overview 

•  FY07  Accomplishments 

•  FY08  Plan 

•  Concluding  Remarks 
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Interoperability  /  Net-Ready  KPP 
Testing  Requirement 


r 


K 


;  \ 

“It  is  expected  any  resultant  materiel  solution  will  be  verified  through 

testing  conducted  in  the  expected  joint  operational  environment  to 
demonstrate  joint  interoperability  and,  when  appropriate,  net-readiness ” 

CJCSI  31 70.01  F,  dated  1  May  2007 


•  DoD  Policy  requires  Joint  interoperability  and  net- 
readiness  testing  during  acquisition 


•  Interoperability  and  Net-Ready  KPP  testing  requires 
testing  interactions  of  multiple  systems  at  the  same 
time 

-  Systems  or  their  representations  are  not  all  co-located 

-  Need  to  test  early  and  throughout  system  development  process 

•  Transition  to  the  GIG  to  realize  Net-Centric  Warfare 
will  increase  the  requirement  for  interoperability  and, 
thus,  increase  the  need  for  distributed  testing 


tAT 


Interoperability,  Net  Ready  Testing 

Challenges 


•  No  “live”  system  available  to  test  early  in  acquisition  process 


•  Other  systems  needed  to  test  interoperability/NR  are  in  various 
stages  of  development 


•  Existing  (legacy)  systems  needed  for  test  are  often  unavailable 
due  to  real  world  commitments  or  too  expensive  to  be  made 
available 


•  Available  “live”  systems  and  system  representations  are 
scattered  across  the  country  on  ranges,  in  integration 
laboratories,  in  simulations,  and  in  other  forms 

•  Difficult,  time-consuming,  and  expensive  to  plan  and  execute 
distributed  test  events 

-  Networks  require  time-consuming  security  agreements  to  be 
coordinated 

-  Instrumentation  data  definitions  differ  from  laboratory  to  laboratory 

-  Lack  of  universal  tools  complicates  test  integration 

-  Distributed  test  events  require  engineering  each  and  every  time 


•  A  corporate  approach  for  linking  distributed  facilities 

-  Enables  customers  to  efficiently  evaluate  their  warfighting 
capabilities  in  a  joint  context 

-  Provides  compatibility  between  test  and  training 

•  A  core,  reusable,  and  easily  reconfigurable  infrastructure 

-  Consists  of  the  following  products: 

•  Persistent  connectivity 

•  Middleware 

•  Standard  interface  definitions  and  software  algorithms 

•  Distributed  test  support  tools 

•  Data  management  solutions 

•  Reuse  repository 

•  Provides  customer  support  team  for  JMETC  products  and 
distributed  testing 
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JMETC  Will  Provide  Infrastructure 

Capability  for: 


•  Testing  across  full  spectrum  of  acquisition  process 


-  Developmental  Test,  Operational  Test 

-  Interoperability  Certification 

-  Net-Ready  KPP  compliance 


•  Joint  mission  portfolio  testing 


•  Evaluation  of  warfighting  capabilities  in  joint  mission 
environment 


•  Conduct  of  live,  virtual  or  constructive  testing 


•  Conduct  of  joint  testing  and  training 

Used  whenever  you  need  to  link  resources  together 
to  conduct  a  distributed  test  event 
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JMETC  Enables 
Distributed  Testing 


Joint  Operational  Scenarios 


Systems 

Under 

Test 

Integrated 

Test 

Resources 


— 
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Virtual 

Prototype 
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Definitions 
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Common 

Middleware 


Hardware 
in  the 
Loop 
Lab 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Installed 

Systems 

Test 

Facility 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Range 


TENA 

Standard 

Interface 

Definitions 
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Common 

Middleware 


JMETC 
Infrastructure 


Reuse 

Repository 


Distributed  Test 
Support  Tools 


Environment 

Generator 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Threat 

Systems 


TENA 

Standard 

Interface 
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Common 
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Data  Management 
Solutions 


Customer  Support 


TEN  A  Architecture  Overview 


JMETC 


TENA  Applications 


Object  Model 
Utilities 


Non-TENA  Non-TENA 


System 


System 


•  •• 


TENA  Utilities 


Non-TENA  Applications 
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JMETC  Leadership  &  Governance 

JMETC 


Chain  of  Command 


JMETC  Accomplishments  -  FY07 

Summary 


•  Supported  two  major  distributed  test  events 

-  Integral  Fire  07 

-  InterTEC  Spiral  2  Build  1 

•  Stood  up  the  JMETC  VPN  on  the  SDREN 

-  Established  8  locations  on  the  JMETC  VPN  available  for  future  use 

•  Pax  River,  C2TF  (@  Eglin),  GWEF  (@  Eglin),  White  Sands,  Redstone, 
China  Lake,  Pt.  Mugu,  and  JITC 

•  Initiated  collaboration  with  the  Training  community 

-  Used  the  JNTC-sponsored  network  aggregator  in  Integral  Fire  07 

-  Supported  the  JFCOM  LVC  Architecture  Roadmap  Study 

•  Established  JMETC  Advisory  Group  and  JMETC  Users 
Group 

•  Conducted  a  DoD  Distributed  Test  Infrastructure  Assessment 

-  Approved  by  the  Joint  Capabilities  Board  (JCB) 
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Integral  Fire  07  Test  Event 


-  Integral  Fire  07  Description: 

•  A  combined,  distributed  test  event  conducted  in  August  07  supporting 
the  following  three  customers: 

-  JFCOM  JSIC  JCAS  Assessment 

-  JTEM  Methodology  Assessment 

-  USAF  Warplan-Warfighter  Forwarder  (WWF) 

-  JMETC  Responsibilities: 

•  Overall  lead  for  creating  the  distributed  test  Infrastructure  including 
JMETC  VPN  (5  locations) 

•  Connect  three  enclaves  (total  of  15  locations)  using  the  JFCOM 
aggregator  router 

•  Conduct  systems  integration,  site  surveys,  and  dry  runs 

•  Oversee  operation  of  the  network  and  data  flow  among  all  sites  during 


the  event 


-  JMETC  Significant  Accomplishments: 

•  Stood  up  and  successfully  demonstrated  the  JMETC  VPN  within  90  days 

•  Successfully  used  the  Aggregation  Router  to  link  three  enclaves 

•  Supported  three  customers  conducting  tests  using  the  same  network  in 
the  same  time  frame 
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Network  Aggregation 
Bridging  Networks 


Aggregation  Router 


JSIC 

Suffolk 


NCTSI 
San  Diego 


CTSF 
Ft  Hood 


SPAWAR 

Charleston 


WARCAP, 

Pentagon 


Pax  River 


RTTC 

Huntsville 

JMETC  VPN 
on  SDREN 


AOC,  Langley  AFB 


SIMAF,  WP  AFB 


AV-8B  Lab 
China  Lake 

F/A-18  Lab 

China  Lake 

IRCC 

WSMR 

GWEF 
Eglin  AFB 


46TW 
Eglin  AFB 


Integral  Fire  07 
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InterTEC  Spiral  2,  Build  1  Test  Event 

(FY07) 


JMETC 


-  Interoperability  T&E  Capability  (InterTEC)  Description: 

•  OSD-sponsored,  Navy-led  project  under  the  Central  T&E  Investment 
Program  (CTEIP) 

•  Purpose  is  to  develop  an  accredited  test  capability  to  conduct  joint 
interoperability  certification  and  joint  mission  thread  testing 


-  Spiral  2,  Build  1  Objectives: 

•  Developing  and  assessing  tools  to  test  joint  threads 

•  Assessing  the  C2  messages  sent  from  sensors  to  shooters  through 
command  and  control  systems  (GCCS-J,  GCCS-M,  GCCS-A,  ana 
TBMCS) 


-  JMETC  Responsibilities: 

•  Overall  lead  for  creating  the  Infrastructure  integrating  6  locations 

•  Conduct  systems  integration,  site  surveys,  and  dry  runs  in  preparation 
for  the  event 

•  Oversee  operation  of  the  network  and  data  flow  among  all  sites  during 
the  event 


-  JMETC  Significant  accomplishments 

•  Established  the  new  locations  on  the  JMETC  VPN  within  90  days 

•  Demonstrated  re-use  (three  locations  from  Integral  Fire  07  test) 

•  Successfully  used  the  Aggregation  Router 


FY  08  Plan 

Event  Support 

-  SIAP  Risk  Reduction  (March  08) 

•  Risk  reduction  test  for  a  planned  Oct  08  event 

-  InterTEC  Spiral  2,  Build  2  (June  08) 

•  Test  OTH-G  messages  using  a  Joint  Fires  Scenario 

•  Integrating  12  locations 

•  Includes  CVN-21  participation 

-  FCS  Combined  Test  Organization  /  JTEM  Test  Event  (July  08) 

•  Test  the  JTEM  Methods  and  Processes 

•  Experiment  and  test  of  the  infrastructure  needed  to  evaluate  joint 
functionality  of  FCS 

-  InterTEC  Systems  Acceptance  Test  (August  08) 

•  JITC  acceptance  test  of  InterTEC  tools 

Collaboration  with  Training  Community 

-  Common  distributed  test  and  training  infrastructure  requirements 

-  JFCOM-led  LVC  Architecture  Roadmap  Study 

-  Demonstration  of  JTEN  and  JMETC  VPN  peering  capabilities 

Support  Other  JM ETC-related  Activities 

-  M&S  Steering  Committee 

-  Distributed  Test  Infrastructure  Studies 
Publish  the  JMETC  Program  Plan 
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Virtual 
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— 

For  this  event:  19  sites 

18  VPN  and  1  via  Network  Agg 
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Joint  Battlespace  Dyn.  Decon  (FCS  &  JTEM) 
JMETC  VPN  (Jul  28-Aug  1,  2008) 
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JMETC  VPN 
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For  this  event:  9  sites 

7  VPN  and  2  via  Network  Agg 
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InterTEC  System  Acceptance  Test 
JMETC  VPN  (Aug  18-29,  2008) 
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Single  Integrated  Air  Picture  (SIAP)  -  JCHE-5 
JMETC  VPN  (1st  Qtr  FY2009) 


SIAP  Attributes: 


*  JROC  Approved  Requirements 

**  SIAP  Measures  Of  Performance 


O  Army 
O  Air  Force 
O  Navy 
•  Marines 
O  Joint 
O  Industry 


■  Completeness 

*  Clarity 

*  Continuity 

*  Kinematic  Accuracy 

*  ID  Completeness 

■  ID  Accuracy 

■  iD  Clarity 

*  Commonality, 


For  this  event:  8  sites 

All  8  sites  on  the  JMETC  VPN 


JMETC  VPN 

via  Network  Aggregator 


After  this  event 

Number  of  Sites 
on  JMETC  VPN 

Times  a  Site  has 
been  Reused 

JMETC  VPN 

24+2 
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Boeing,  Seattle 
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Projected  JMETC  Connectivity 
(at  end  of  FY08) 


JMETC  Connectivity  Status  (end  of  FY08) 

Dedicated,  trusted  VPN  on  SDREN  (part  of  the  GIG) 
Encrypted  for  Secret  -  System  High 

26  sites  based  on  customer  needs  &  potential  for  reuse 

•  Includes  sites  at  industry  facilities 

Connected  to  JNTC-sponsored  Network  Aggregator 

Capable  to  bridge  to  sites  on  other  classified  networks 

•  JNTC JTEN 

•  DISN  SIPRNET 

•  AF-ICE  Enclave 

•  Other  unique  classified  enclaves  on  DREN 


O  Army 
O  Air  Force 
O  Navy 
•  Marines 
O  Joint 
O  Industry 


TCC  &  EA-6, 
Pt  Mugu 

Camp  Pendleton 


ICSTF  &  RLBTS, 
Pt  Loma 


C2TF  &  GWEF, 
Eglin  AFB 


JMETC  VPN 

via  Network  Aggregator 


At  end  of  FY08 

Number  of  Sites 
on  JMETC  VPN 

Times  a  Site  has 
been  Reused 

JMETC  VPN 

26 

46 

In  the  First  Year  JMETC  Has: 


•  Stood  up  the  VPN 

•  Supported  two  major  events 

-  Four  customers 

-  Seven  instances  of  site/security  agreement  reuse 

•  Aggregated  three  network  enclaves 

-  Leveraging  Service  established  sites/security  agreements  when 
ever  possible 

•  Begun  event  planning  with  four  major  acquisition  programs 

-  CVN-21,  FCS,  JSF,  SIAP 

-  Indicates  that  they  each  see  value  in  JMETC 

•  Won  the  cooperative  and  collaborative  support  of  the 
Services  and  Agencies 

-  Our  success  is  dependent  on  their  future  support 

•  Conducted  the  Distributed  Test  Infrastructure  Study 

-  Resulted  in  three  gaps  to  be  analyzed  in  three  study  tasks 

•  Next  slide 
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Distributed  Test  Infrastructure 

Studies 

•  FY07  Distributed  Test  Infrastructure  Assessment,  approved  by  the  JCB, 
resulted  in  need  for  three  studies: 

-  Task  1:  Transition  from  IPv4  to  IPv6  at  test  facilities  and  laboratories 

•  Scope:  What  is  the  modernization  schedule  of  each  of  the  Services  to  IPv6  at 
their  test  facilities  and  laboratories? 

-  Task  2:  Applicability  of  Service-Oriented  Architectures  (SOA)  to 
Distributed  Testing  Infrastructure 

•  Scope:  When  will  SOAs  be  suitable  to  support  distributed  testing  data 
management  requirements? 

-  What  are  the  benefits  of  modernizing  instrumentation  to  use  a  SOA  for  testing? 

-  What  are  the  benefits  of  modernizing  distributed  test  tools  to  use  a  SOA  for  testing? 

-  Task  3:  Test  Infrastructure  Required  for  Warfighting  Systems  using  the 
Global  Information  Grid  (GIG) 

•  Scope:  What  future  instrumentation,  distributed  test  tools,  connectivity,  and  data 
management  capabilities  will  be  needed  to  conduct  distributed  tests  to  verify 
warfighting  capabilities  (in  CT,  DT,  OT,  etc.)  are  operating  correctly  with  the  GIG? 

•  Studies  to  be  completed  by  April  FY09 


JMETC  Program  Points  of  Contact 


JMETC  Program  Manager: 


Chip  Ferguson 

chip.ferquson@osd.mil 

703-604-0350x1 38 


JMETC  Lead  Systems  Engineer: 


George  Rumford 

qeorqe.rumford@osd.mil 

703-601-5233 


JMETC  Operations,  Planning 
Support: 


Len  Zimmermann 

leonard.zimmermann.ctr@osd.mil 

703-604-0350x141 


JMETC  Program  Office  Contact: 

-  E-mail:  imetc-feedback@imetc.org 

-  Telephone:  (703)  604-0350  ext.  0 

-  JMETC  Website:  www.imetc.org  -  under  construction 

-  JMETC  Help  Desk:  www.imetc.org  -  under  construction 

TENA  Website:  www.tena-sda.org 

-  Links  to  JMETC,  Help  Desk,  and  products 
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Backup 


Summary 


•  Provide  for  the  full  spectrum  of  Joint  testing,  supporting 
many  customers  in  many  different  Joint  mission  threads 

•  Being  built  based  on  current  customer  requirements 
-CVN-21,  SIAP,  FCS,  JSF,  MMA,  NECC,  DD1000,  WWF 

•  Partnering  with  Service  and  Agency  activities 
-Leveraging  existing  capabilities 

•  Working  with  JFCOM  to  develop  a  joint,  multi-use  test 
and  training  network  infrastructure 
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Background 


•  March  2004  -  SPG:  “Joint  Testing  in  Force  Transformation” 

-  Policy  -  Developing  and  fielding  joint  force  capabilities  requires  adequate , 
realistic  test  and  evaluation  in  a  joint  operational  context 

-  Direction  -  DoD  will  provide  new  testing  capabilities  and  institutionalize  the 
evaluation  of  joint  system  effectiveness 

-  Action  -  DOT&E  lead  development  of  a  Roadmap  to  define  changes  to 
ensure  that  T&E  is  conducted  in  a  joint  environment  and  facilitates  the 
fielding  of  joint  capabilities 

•  November  2004  -  DEPSECDEF  approved  Roadmap,  validated  SPG 

•  December  2005  -  Department  directed  stand-up  of  the  Joint  Mission 
Environment  Test  Capability  (JMETC)  Program  Element  under 
USD(AT&L)/  TRMC  for  execution 

•  October  2006  -  Establishment  of  JMETC  Program  Management 
Office  in  Crystal  City,  VA 


JMETC  is  <1.5  years  old 


25 


JMETC  Benefits 


JMETC 


•  Provides  Department-wide  capability  for: 


-  Evaluation  of  a  weapon  system  in  a  joint  context 

-  DT,  OT,  Interoperability  Certification,  Net-Ready  KPP  compliance  testing, 
Joint  Mission  Capability  Portfolio  testing,  etc. 

-  Effectively  and  efficiently  linking  distributed  test  facilities 

-  More  robust  testing  earlier  in  the  acquisition  process 

-  Improved  system  interoperability 

•  Provides  test  capability  aligned  with  JNTC 

-  Both  use  TENA  architecture  to  integrate  resources 

-  Enables  joint  test  and  training 

•  Reduces  time  and  cost  by  providing 

-  Readily  available,  persistent  connectivity  with 
standing  network  security  agreements 

-  Common  integration  software  for  linking  sites 

-  Distributed  test  planning  support  tools 

•  Provides  distributed  test  expertise 


The  warnghtar  is  the  u  Him  aria  hw&ficlary  with 
warngMlng  capabilities  vanned  to  work  together 
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Combinatorial  Testing: 

Required  Knowledge  in  a  World  Where  Traversing 
Every  Possible  Path  is  Impossible 


Dr.  Mark  J.  Kiemele 
Air  Academy  Associates 


24th  Annual  National  Test  and  Evaluation  Conference 

Palm  Springs,  CA 
February  25,  2008 


Introductions 


■  Name 

■  Organization 

■  Job  Title/Duties 

■  Experience  in  T&E,  Combinatorial 
Testing,  etc. 
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Agenda 

■ 

Some  Basic  Definitions 

■ 

Various  Approaches  to  Testing  Multiple  Factors 

■ 

Design  of  Experiments  (DOE):  a  Modern  Approach  to 
Combinatorial  Testing 

■ 

Break 

■ 

Examples  and  Demonstration  of  a  DOE 

■ 

Using  DOE  to  Achieve  Robust  Designs 

■ 

DOE  with  Modeling  and  Simulation 

■ 

High  Throughput  Testing 
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Definition  of  a  Process 


INPUTS 

OUTPUTS 

x, - - 

V 

PROCESS 

1 

v 

A3 - -m- 

i- Y" 

A  blending  of 

x5 - - 

inputs  to 

achieve  some 

x6 - * 

desired  outputs 

- -Y3 

x7 - - 

Acadlmy 
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Simplify,  Perfect,  Innovate 


Graphical  Meaning  of  y  and  a 


y  =  Average  =  Mean  =  Balance  Point 
a  =  Standard  Deviation 


Concave  Down 


t£F 

Academy 

Associates 

Simplify,  Perfect,  Innovate 


o  ~  average  distance  of  points 
from  the  centerline 
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Graphical  View  of  Variation 


i-  68.27% 


-  95.45%  - 

i -  99.73%  - 

-  99.9937% 

-  99.999943% 

99.9999998% 


Academy 


Typical  Areas  under  the  Normal  Curve 
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Simplify,  Perfect,  Innovate 


Approaches  to  Testing  Multiple  Factors 


*  Traditional  Approaches 

•  One  Factor  at  a  Time  (OFAT) 

•  Oracle  (Best  Guess) 

•  All  possible  combinations  (full  factorial) 

*  Modern  Approach 

•  Statistically  designed  experiments  (DOE) ...  full 
factorial  plus  other  selected  DOE  designs, 
depending  on  the  situation 
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OFAT  (One  Factor  at  a  Time) 


X1  =  time 

Yield  (gr.)  

► 

Chemical 

X2  =  temp 

Process 

w 

1 .  Hold  X2  constant  and  vary  X1 

Find  the  “best  setting”  for  X., 
Y 


60  90  120  150 


2.  Hold  X1  constant  at  “best  setting”  and  vary  X2. 
Find  the  “best  setting”  for  X2. 

Y 


> 


A,* 

Academy 

A  . 


Simplify,  Perfect,  Innovate 


3.  One  factor  at  a  time 
results 


4.  One  factor  at  a  time  results 


versus  optimal  results 
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Oracle  (Best  Guess) 


W  =  Wetting  Agent  (1  =.07  ml;  2=none) 

P  =  Plasticizer  (1=1  ml;  2=none) 

E  =  Environment  (1=Ambient  Mixing;  2=Semi-Evacuated) 
C  =  Cement  (1=Portland  Type  III;  2=Calcium  Aluminate) 

A  =  Additive  (1=No  Reinforcement;  2=Steel) 

Y  =  Strength  of  Lunar  Concrete 


f 

V] 


Academy 

A  . 


Run 

W 

P 

E 

C 

A 

Y 

1 

1 

2 

1 

1 

1 

5 

2 

1 

1 

1 

1 

1 

6 

3 

2 

2 

1 

1 

1 

5 

4 

2 

1 

1 

1 

2 

6 

5 

1 

2 

2 

2 

2 

7 

6 

1 

1 

2 

2 

2 

8 

7 

2 

2 

2 

2 

2 

10 

8 

2 

1 

2 

2 

1 

11 
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Simplify,  Perfect,  Innovate 


\ 


Academy 

Associates 


Simplify,  Perfect,  Innovate 


Evaluating  the  Effects  of  Variables  on  Y 


What  we  have  is: 


What  we  need  is  a  design 
to  provide  independent 
estimates  of  effects: 


How  do  we  obtain  this  independence  of  variables? 
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Example  1 : 

A  (2  levels) 
B  (2  levels) 


All  Possible  Combinations 
(Full  Factorial) 


MATRIX  FORM 


TREE  DIAG RAM 


A  B 


1 

1 

2 

2 


1 

2 

1 

2 


\ 


Academy 

Associates 


Simplify,  Perfect,  Innovate 


Ex  a  m  p  I  e  2:  A 


A  (3  levels)  1 

B  (3  levels)  1 

C  (2  levels)  *1 


2 

2 

2 

3 

3 

3 

1 

1 

1 

2 

2 

2 

3 

3 

3 
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B 


1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 

1 

2 

3 
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0V  V  V V  V  VV  V  V 

w  /-v  i  no  _  m  m  _  m  rr> 


Design  of  Experiments  (DOE) 


•  An  optimal  data  collection  methodology 

•  “Interrogates”  the  process 

•  Used  to  identify  important  relationships 
between  input  and  output  factors 

•  Identifies  important  interactions  between 
process  variables 

•  Can  be  used  to  optimize  a  process 

•  Changes  “I  think”  to  “I  know” 
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Important  Contributions  From: 


TAGUCHI 

SH  AIN  IN 

CLASSICAL 

BLENDED 

APPROACH 

Loss  F u notion 

* 

* 

Emphasis  on  Variance 

* 

* 

Reduction 

Robust  Designs 

* 

* 

KISS 

* 

* 

* 

Sim  pie  Significance 

* 

* 

T  ests 

Component  Swapping 

* 

M  ultivariate  Charts 

* 

* 

M  odeling 

* 

* 

Sam  pie  Size 

* 

* 

Efficient  Designs 

* 

* 

0  ptim  ization 

* 

* 

Confirm  ation 

* 

* 

Response  Surface 

M  ethods 

* 

* 

Simplify,  Perfect,  Innovate 


Which  bag  would  a  world  class  golfer  (prefer? 
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Statistically  Designed  Experiments  (DOE) 

Orthogonal  or  Nearly  Orthogonal  Designs 


FULL  FACTORIALS  (for  small  numbers  of  factors) 
FRACTIONAL  FACTORIALS 
PLACKETT  -  BURMAN  ^ 

LATIN  SQUARES  ►  Taguchi  Designs 

HADAMARD  MATRICES 
BOX  -  BEHNKEN  DESIGNS 
CENTRAL  COMPOSITE  DESIGNS 

SIMPLE  DEFINITION  OF  TWO-LEVEL 
ORTHOGONAL  DESIGNS 


1 

\ 


A,* 

Academy 


Run 

Actual  Settings 

(5,10)  (70,90) 

A:  Time  B:  Temp 

(100,  200) 

C:  Press 

(A) 

Time 

Coded  Matrix 

(B) 

Temp 

(C) 

Press 

Responses 

1 

5 

70 

100 

-1 

-1 

-1 

2 

5 

70 

200 

-1 

-1 

+  1 

3 

5 

90 

100 

-1 

+  1 

-1 

4 

5 

90 

200 

-1 

+  1 

+  1 

5 

10 

70 

100 

+  1 

-1 

-1 

6 

10 

70 

200 

+  1 

-1 

+  1 

7 

10 

90 

100 

+  1 

+  1 

-1 

8 

10 

90 

200 

+  1 

+  1 

+  1 
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Simplify,  Perfect,  Innovate 


The  Beauty  of  Orthogonality: 

independent  evaluation  of  effects 


A  Full  Factorial  Design  for  3  Factors,  Each  at  2  Levels 
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Building  a  Fraction  of  the  Full  Factorial 


Given  the  number  of  factors  (k)  =  4 


\ 


A,* 

Academy 

ASSOCIATES 


Simplify,  Perfect,  Innovate 


i)  The  full  factorial  has  n  =  24  =  16  runs 

ii)  A  V2  fraction  will  have  n  =  24/2  =  24'1  =  8  runs 

STEP  1 :  Build  an  8  run  design  for  3  factors:  A,  B,  and  C 

STEP  2:  Alias  (perfectly  confound)  the  4th  factor,  D,  with  the  highest  order  interaction  in  Step  1 

STEP  3:  Determine  Aliasing  (confounding)  Pattern 


A 

B 

A  B 

C 

A  C 

B  C 

D  =  A  B  C 

Run 

1 

2 

3 

4 

5 

6 

7 

1 

- 

- 

+ 

- 

+ 

+ 

- 

2 

- 

- 

+ 

+ 

- 

- 

+ 

3 

- 

+ 

- 

- 

+ 

- 

+ 

4 

- 

+ 

- 

+ 

- 

+ 

- 

5 

+ 

- 

- 

- 

- 

+ 

+ 

6 

+ 

- 

- 

+ 

+ 

- 

- 

7 

+ 

+ 

+ 

- 

- 

- 

- 

8 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

A  v  g  (-) 

A  v  g  (+  ) 

A 

A  /  2 

2 5-1  DESIGN  EXAMPLE 


Run 

Factors 

DE  CE  BE  AE  E 

A  B  C  D  AB  AC  BC  AD  BD  CD  ABC  ABD  ACD  BCD  ABCD 

Response 

Vi  y2  y  3 

1 

2 

_ 16 _ 
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DOE  Helps  Determine  How  Inputs  Affect 

Outputs 


i)  Factor  A  affects  the  average  of  y 


y 

ii)  Factor  B  affects  the  standard  deviation  of  y 


\ 


& 

Academy 

Associates 


y 

iii)  Factor  C  affects  the  average  and  the 
standard  deviation  of  y 
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Simplify,  Perfect,  Innovate 


KISS  Guidelines  for  Choosing  an 
Experimental  Design 


Academy 


Nb! 


KISS  -  Keep  It  Simple  Statistically 


Source: 


Steve  Schmidt 
Mark  Kiemele 
Cass  Grandone 


^  start  ^ 


STATEMENT 
of  the 
PROBLEM 
& 

OBJECTIVE 

5 


PF 


CE 


Process  flow  diagram 


y-c 


Cause-and-effect  diagram 


outputs 


Inputs  on  cause-and-effect  to  be  held  constant 

Inputs  on  cause-and-effect  that  are  noise  or  uncontrolled 

Inputs  (factors)  on  cause-and-effect  identified  for  experimentation 

Standard  operating  procedures  to  insure  all  Cs  are  held  constant 
and  process  flow  is  complied  with 


FULL  FACTORIAL 

K  =  2  .. 

■  nreps>9 

K  =  3  .. 

•  "reps  ^  5 

K  =  4  ., 

-  Teps-  3 

25-1 

y2  FRACTION 


12  Run 

PLACKETT-BURMAN 
or  TAGUCHI  L12 
SCREENING 


nrftns  -  4 


16  Run 

FRACTIONAL 

FACTORIAL 

"reps  ^  3 


FULL  FACTORIAL 


K  =  2  ...  nreps  >  7 
K  -  3  ...  nreps  >  3 


SCREENING 
(also  includes  One 
2-level  Factor) 


n  >4 

1  'mps  ~  ^ 


Simplify,  Perfect,  Innovate 


NOTE  1 :  Sample  size  (nreps)  is  for  95%  confidence  is  and  99.99%  confidence  iyi  . 

NOTE  2:  (nreps/2)  will  provide  75%  confidence  ir$  and  95%  confidence  iyi  . 

NOTE  3:  The  12  Run  Plackett-Burman  or  LI  2  is  very  sensitive  to  large  numbers  of  interactions.  If  this  is  the  case,  you  would 
be  better  off  using  the  16  Run  Fractional  Factorial  or  a  smaller  number  of  variables  in  2  or  more  full  factorial 
experiments. 

NOTE  4:  For  more  complete  2-level  design  options,  see  next  page. 


CENTRAL  COMPOSITE 

or 

BOX-BEHNKEN  DESIGN 

K  =  2  .. 

•  nrepS  -  9  (CCD) 

K  =  3  .. 

■  nreps  >  5  (CCD  or  BB) 

K  =  4  .. 

.  nreps>3(CCD  or  BB) 

K  =  5  .. 

.nreps>3(CCD) 
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Building  a  Screening  Design 


Li  2  Design 

Run 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

1 

2 

■ 

■ 

- 

- 

+ 

+ 

+ 

+ 

+ 

+ 

3 

- 

+ 

+ 

+ 

- 

- 

- 

+ 

+ 

+ 

4 

+ 

- 

+ 

+ 

- 

+ 

+ 

- 

- 

+ 

5 

+ 

+ 

- 

+ 

+ 

- 

+ 

- 

+ 

- 

6 

+ 

+ 

+ 

- 

+ 

+ 

- 

+ 

- 

- 

7 

+ 

- 

+ 

+ 

- 

- 

+ 

+ 

- 

+ 

- 

8 

+ 

- 

+ 

- 

+ 

+ 

+ 

- 

- 

- 

+ 

9 

+ 

- 

- 

+ 

+ 

+ 

- 

+ 

+ 

- 

- 

10 

+ 

+ 

+ 

- 

- 

- 

- 

+ 

+ 

- 

+ 

11 

+ 

+ 

- 

+ 

- 

+ 

- 

- 

- 

+ 

+ 

12 

+ 

+ 

- 

- 

+ 

- 

+ 

- 

+ 

+ 

- 

t£F 

Academy 

Associates 
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Simplify,  Perfect,  Innovate 


Transfer  Functions 


r 


Parameters 
or  Factors^ 
that 

Influence 
the  CTC 


y  =fi(x1,x2Ix3) 
s  =f2(x1,x2,  x3) 


Where  does  the  transfer  function  come  from? 

•  Exact  transfer  Function 

*  Approximations 


t£F 

Academy 

Associates 

Simplify,  Perfect,  Innovate 


■  DOE 

-  Historical  Data  Analysis 
Simulation 
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Exact  Transfer  Function 


•  Engineering  Relationships 
-  V  =  IR 
F  =  ma 


Ri 


AAAA 


Ro 


The  equation  for  the  impedance  (Z) 
through  this  circuit  is  defined  by: 


z  = 


R1  R2 

R1  +  R2 


The  equation  for  magnetic  force  at  a  distance 
X  from  the  center  of  a  solenoid  is: 

.5f  +  x  ^  .5f-x 

>2+(.5^  +  x)2  +>2+(.5^-x)2_ 


t£F 

Academy 

Associates 

Simplify,  Perfect,  Innovate 


Where 


N  :  total  number  of  turns  of  wire  in  the  solenoid 
l  :  current  in  the  wire,  in  amperes 
r  :  radius  of  helix  (solenoid),  in  cm 
£ :  length  of  the  helix  (solenoid),  in  cm 
x :  distance  from  center  of  helix  (solenoid),  in  cm 
H  :  magnetizing  force,  in  amperes  per  centimeter 
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Catapulting  Power  into  Test  and  Evaluation 


f 

\ 


A1,I* 

Academy 

Associates 


Statapult®  Catapult 
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Simplify,  Perfect,  Innovate 


The  Theoretical  Approach 


y 


\ 


Ah?* 

Academy 
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Simplify,  Perfect,  Innovate 


The  Theoretical  Approach  (cont.) 


Io0  =  rFF(9)sin0coscp-(MgrG  +  mgrB)sin0 

0 


tan  <|)  = 


D-rp  sin0 
d  +  rF  cos  0  ’ 


1 


lo0  =  rF  F(0)sin0  coscpd0-(MgrG  +mgrB)(sin0-sin0o) 


1 


■I 

% 

6b 


lo02  =  rF  F(0)sin0coscpd0-(MgrG  +mgrB)(sin01  -sin0o). 


x  =  vB  cos 


nY  1 


-01  t - TgCOS©! 


y  =  rBsin01  +  vBsin 


j 


^-e,)  1 

V2  ') 


t--gt2. 
2y 


A,k^ 

Academy 


rB  sin0-| +(R  +  rB  cos0-|)tan  — 0-j 


g  (R  +  rsCos©^' 


^  J  2V|cos2 


;§v 


=  0. 


gio 


(R  +  rB  cos  0-|  Y 


4rB  2 
cos 


(f-e.) 


rB  sin01  +(R  +  rB  cos  0^  tan  — 01 

v2  J 


9| 


=  r(=j*  F(0)sin0  cos4>de-(M8iG  +  mgrB)(sin91  -sine„). 
% 
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Simplify,  Perfect,  Innovate 


Statapult®  DOE  Demo 

(The  Empirical  Approach) 


A.k'* 

Academy 


rasKnigiisa 


Actual 

Factors 


Coded  Factors 


Response  Values 


Run 

A 

B 

A 

B 

AB 

Y,  Y2 

Y 

S 

1 

144 

2 

-1 

-1 

+1 

2 

144 

3 

-1 

+1 

-1 

3 

160 

2 

+1 

-1 

-1 

4 

160 

3 

+1 

+1 

+1 

Avg  - 

Avg  + 

A 
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Simplify,  Perfect,  Innovate 


Minimizing  the  #  of  Factor  Changes 


(GRAY  CODE  SEQUENCE) 

Problem:  If  changing  factor  settings  is  time  consuming  and/or  expensive,  using  a  Gray 
Code  sequence  to  determine  the  sequence  of  runs  may  be  useful.  A  Gray  Code 
sequence  orders  the  runs  so  that  only  1  factor  setting  changes  between  runs 
and  the  most  difficult  to  change  factors  are  changed  less  frequently. 


16-Run  Design 


Academy 


Gray  Code  by  Run  # 
1 
2 

4 
3 

7 

8 
6 

5 

13 

14 
16 

15 
11 
12 
10 
9 
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Cycling  through  the  runs  from  top  to  bottom  (or 
vice  versa)  will  produce  15  changes: 

•  D  will  be  changed  8  times. 

•  C  will  be  changed  4  times. 

•  B  will  be  changed  2  times. 

•  A  will  be  changed  1  time. 

Thus,  the  most  difficult  (or  expensive)  to  change 
factors  should  be  assigned  to  A,  B,  C,  D, 
respectively. 
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Simplify,  Perfect,  Innovate 


\ 


Academy 
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Simplify,  Perfect,  Innovate 


Test  Sequence  Generator 


Gray  Code  Sequence  Generator  (Wheel) 
by  Run  Number  for  16  Runs  and  4  Factors 

©2008  Air  Academy  Associates,  LLC.  Do  Not  Reproduce.  Page  27 


Value  Delivery:  Reducing  Time  to  Market  for 

New  Technologies 


INPUT 


OUTPUT 


Pitch  <) 

(0,  15,  30) 

Roll  <) 

(0,  15,30)  J 

W1F  <) 

(-15,0,  15)  J 

W2F  <) 

(-15,0,15)  . 

W3F  <) 

(-15,0,  15)  J 

Six  Aero- 


Characteristics 


A.k'* 

Academy 


•  Total  #  of  Combinations  =  35  =  243 

•  Central  Composite  Design:  n  =  30 


Simplify,  Perfect,  Innovate 


Patent  Holder:  Dr.  Bert  Silich 
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Simplify,  Perfect,  Innovate 


Central  Composite  Designs 

(Box-Wilson  Designs) 


CCD  for  K=3  Factors 

FACTOR 

Run 

ABC 

1 

2 

+ 

3 

+ 

F  4 

+  + 

5 

+ 

6 

+  -  + 

7 

+  + 

8 

+  +  +  F  =  factorial  portion 

9 

0  0  0 

C  10 

o  o  o  C  =  centerpoint  portion 

11 

0  0  0 

12 

-a  0  0  A  =  axial  portion 

13 

+a  0  0 

A  14 

0  -a  0 

15 

0  +a  0 

16 

o 

o 

1 

p 

17 

0  0  +a 
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Suggested  Values  for  a  and  Center  Points 

(Central  Composite  Designs) 


Face-centered  Design  (a  =  1) 

-  Hard  limits  (restrictions)  on  factor  settings 

-  Cannot  take  factor  settings  beyond  ±1  (coded  values) 

-  Predictions  made  within  the  “cube” 

-  Recommended  number  of  center  points  =  2 
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A,* 

Academy 
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•  Spherical  Design  (a  =vr  ) 
Rotatable  Design  (a  =  (nF)1/4) 


-  No  hard  limits  (constraints)  on  factor  settings  ^ — j — 

-  Able  to  go  beyond  ±1  coded  settings 

-  Predictions  slightly  beyond  the  “cube”  (in  case  the  optimum  lies  just 
outside) 


-  Orthogonality  can  be  an  issue,  so  a  larger  number  of  center  points  is 
recommended  (between  3  and  6) 


•  nF  is  the  number  of  runs  in  the  factorial  part  of  the  design 

•  k  is  the  number  of  factors 
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Simplify,  Perfect,  Innovate 


Aircraft  Equations 


CL  =  .233  +  ,008(P)2  +  ,255(P)  +  .012(R)  -  ,043(WD1)  -  .1 17(WD2)  +  ,185(WD3)  +  ,010(P)(WD3)  - 

.042(R)(WD1)  +  ,035(R)(WD2)  +  ,016(R)(WD3)  +  ,010(P)(R)  -  ,003(WD1)(WD2)  - 
.006(WD1)(WD3) 

CD  =  .058  +  .01 6(P)2  +  ,028(P)  -  ,004(WD1 )  -  .01 3(WD2)  +  .01 3(WD3)  +  ,002(P)(R)  -  ,004(P)(WD1 ) 

-  ,009(P)(WD2)  +  ,016(P)(WD3)  -  ,004(R)(WD1)  +  ,003(R)(WD2)  +  ,020(WD1)2  +  ,017(WD2)2 
+  .021  (WD3)2 

CY  =  -.006(P)  -  ,006(R)  +  .169(WD1)  -  ,121(WD2)  -  ,063(WD3)  -  ,004(P)(R)  +  ,008(P)(WD1)  - 

.006(P)(WD2)  -  ,008(P)(WD3)  -  ,012(R)(WD1)  -  ,029(R)(WD2)  +  ,048(R)(WD3)  -  ,008(WD1)2 

CM  =  .023  -  .008(P)2  +  ,004(P)  -  ,007(R)  +  ,024(WD1 )  +  ,066(WD2)  -  ,099(WD3)  -  ,006(P)(R)  + 

.002(P)(WD2)  -  .005(P)(WD3)  +  ,023(R)(WD1)  -  ,019(R)(WD2)  -  ,007(R)(WD3)  +  ,007(WD1)2 

-  .008(WD2)2  +  .002(WD1)(WD2)  +  ,002(WD1)(WD3) 

CYM=  .001  (P)  +  .001  (R)  -  .050(WD1 )  +  ,029(WD2)  +  .01 2(WD3)  +  .001  (P)(R)  -  ,005(P)(WD1 )  - 

.004(P)(WD2)  -  .004(P)(WD3)  +  ,003(R)(WD1)  +  ,008(R)(WD2)  -  ,013(R)(WD3)  +  ,004(WD1)2 
+  ,003(WD2)2  -  .005(WD3)2 


Acadlmy 


Ce  =  ,003(P)  +  ,035(WD1 )  +  ,048(WD2)  +  .051  (WD3)  -  .003(R)(WD3)  +  .003(P)(R)  -  .005(P)(WD1 ) 

+  ,005(P)(WD2)  +  .006(P)(WD3)  +  ,002(R)(WD1) 
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Simplify,  Perfect,  Innovate 


Fusing  Titanium  and  Cobalt-Chrome 


1 


\ 
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Simplify,  Perfect,  Innovate 


Courtesy  Rai  Chowdhary 
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Hierarchical  Transfer  Functions 


Y  =  Gross  Margin  = 


Gross  Profit 
Gross  Revenue 
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Y  =  y2,  y3,  y4,  y5,  y6) 


Costpost  sales  =  f(field  cost,  remote  services,  suppliers) 


x1  =  f(direct  labor,  freight,  parts,  depreciation) 
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Simplify,  Perfect,  Innovate 


Modeling  The  Drivers  of  Turnover 


©  External  Market  Factors 

(Local  Labor  Market  Conditions) 

Local  Unemployment  Rate 

Local  Employment  Alternatives 

Company’s  Market  Share 


Organizational  Characteristics 
and  Practices 

Supervisor  Stability 

Lateral  /  Upward  Mobility 

Layoff  Climate 


Academy 


©  Employee  Attributes 

Time  Since  Last  Promotion 
Education  Level 
Job  Stability  History 
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Process  of 
Deciding  to 
Stay  /  Leave 
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Turnover  Rate 


*- 


Simplify,  Perfect,  Innovate 


DOE  “Market  Research”  Example 


Suppose  that,  in  the  auto  industry,  we  would  like  to  investigate  the  following  automobile 
attributes  (i.e.,  factors),  along  with  accompanying  levels  of  those  attributes: 


A.k'* 

Academy 


A:  Brand  of  Auto: 

-1  =  foreign 

+1  =  domestic 

B:  Auto  Color: 

-1  =  light 

0  =  bright 

+1  =  dark 

C:  Body  Style: 

-1  =  2-door 

0  =  4-door 

+1  =  sliding  door/hatchback 

D:  Drive  Mechanism: 

-1  =  rear  wheel 

0  =  front  wheel 

1  +1  =  4-wheel 

E:  Engine  Size: 

-1  =  4-cylinder 

0  =  6-cylinder 

+1  =  8-cylinder 

F:  Interior  Size: 

-1  <  2  people 

0  =  3-5  people 

+1  >  6  people 

G:  Gas  Mileage: 

-1  <  20  mpg 

0  =  20-30  mpg 

+1  >  30  mpg 

H:  Price: 

-1  <  $20  K 

0  =  $20-$40K 

+1  >  $40 K 

In  addition,  suppose  the  respondents  chosen  to  provide  their  preferences  to  product 
profiles  are  taken  based  on  the  following  demographic: 

J:  Age: 

-1  <  25  years  old 

+1  >  35  years  old 

K:  Income: 

-1  <  $30K 

+1  >  $40 K 

L:  Education: 

-1  <  BS 

+1  >  BS 
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Simplify,  Perfect,  Innovate 


Robust  Design 


Process  of  finding  the  optimal  location  parameters  (i.e.,  means) 
of  the  input  variables  to  minimize  dpm. 


t 


ACADEMY 


p2 

Simplify,  Perfect,  Innovate 

- w 
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Why  Robust  Design? 


If  nx  varies,  should  we  select  ji1  or  \i2  to  hit  y  =  T? 
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Robust  (Parameter)  Design 
Simulation  Example 


Plug  Pressure  (20-50) 


Bellow  Pressure  (10-20) 

- ► 


Ball  Valve  Pressure  (100-200) 


Water  Temp  (70-100) 


Nuclear 

Reservoir 

Level 

Control 

Process 


Reservoir  Level  (700-900) 


> 


t2F 

Academy 

Associates 
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Simplify,  Perfect,  Innovate 


Examples  of  Simulation  and  High 
Performance  Computing  (HPC) 


Power 


Automotive 


Simulation  of  underhood  thermal  cooling  for  decrease 
in  engine  space  and  increase  in  cabin  space  and  comfort 


Simulation  of  stress  and  vibrations  of  turbine 
assembly  for  use  in  nuclear  power  generation 


1 

\ 
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Aerospace 


Electronics 


Evaluation  of  dual  bird-strike  on  aircraft  engine 
nacelle  for  turbine  blade  containment  studies 


Evaluation  of  cooling  air  flow  behavior 
inside  a  computer  system  chassis 
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Simplify,  Perfect,  Innovate 


Examples  of  Computer  Aided  Engineering 
(CAE)  and  Simulation  Software 


Mechanical  motion:  Multibody  kinetics  and  dynamics 
ADAMS® 

DADS 

Implicit  Finite  Element  Analysis:  Linear  and  nonlinear  statics, 
dynamic  response 

MSC.Nastran™,  MSC.Marc™ 

ANSYS® 

Pro  MECHANICA 

ABAQUS®  Standard  and  Explicit 

ADINA 


Explicit  Finite  Element  Analysis  :  Impact  simulation,  metal 
forming 

LS-DYNA 

RADIOSS 

PAM-CRASH®,  PAM-STAMP 


t£F 
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Simplify,  Perfect,  Innovate 


General  Computational  Fluid  Dynamics:  Internal  and  external 
flow  simulation 
STAR-CD 
CFX-4,  CFX-5 
FLUENT®,  FIDAP™ 

PowerFLOW® 
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Examples  of  Computer  Aided  Engineering 
(CAE)  and  Simulation  Software  (cont.) 


Preprocessing:  Finite  Element  Analysis  and 
Computational  Fluid  Dynamics  mesh  generation 
ICEM-CFD 
Gridgen 

Altair®  HyperMesh® 
l-deas® 

MSC.Patran 

TrueGrid® 

GridPro 

FEMB 

ANSA 


1 

\ 
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Simplify,  Perfect,  Innovate 


Postprocessing:  Finite  Element  Analysis  and 
Computational  Fluid  Dynamics  results  visualization 
Altair®  HyperMesh® 
l-deas 
MSC.Patran 
FEMB 
EnSight 
FIELDVIEW 

ICEM  CFD  Visual3  2.0  (PVS) 

COVISE 
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Multidisciplinary  Design  Optimization  (MDO): 
A  Design  Process  Application 


Design  Responses  (Minimize) 


•  CFD 

•  Structures 

•  Performance 

•  Controls 

•  Cost 


•NVH 

•  Safety 

•  Durability 

•  Controls  Stability 

•  Cost 


Key  Elements  of  MDO 

•  Massive  Computational  Problem; 

•  Solution  by  decomposition  effective  for  complex  systems; 

•  Multiprocessor  computing  simplifies  MDO  solutions 
conceptually  &  enables  solutions  previously  intractable; 

•  Aids  in  the  management  of  the  design  process. 


Mastery  of  interactions  between  the  disciplines  (or,  subsystems) 
is  as  important  as  the  methods  &  tools  used  within  a  single  discipline 
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Simplify,  Perfect,  Innovate 


Latin  Hypercube  Sampling 


•  Method  to  populate  the  design  space  when 
using  deterministic  simulation  models  or  when 
many  variables  are  involved. 


•  Design  space  has  k  variables  (or  dimensions). 
Ex:  Assume  k  =  2 


Suppose  a  sample  of  size  n  is  to  be  taken; 
Stratify  the  design  space  into  nk  cells. 

Ex:  Assume  n  =  5;  nk  =  52  =  25 

Note:  there  are  n  strata  for  each  of  the  k 
dimensions. 


t£F 

Academy 

Associates 

Simplify,  Perfect,  Innovate 


Each  of  the  n  points  is  sampled  such  that  each 
marginal  strata  is  represented  only  once  in  the 
sample. 

Note:  each  sample  point  has  its  own  unique 
row  and  column. 


>  *i 


x2 


x2 


©2008  Air  Academy  Associates,  LLC.  Do  Not  Reproduce. 


Page  44 


Applying  Modeling  and  Simulation  to 
Automotive  Vehicle  Design 


Academy 


IDENTIFY 

Many,  Many  x’s 

SCREENING  DESIGN 

The  critical 

CTCs,  CDPs 

(DOE  PRO) 

few  CDP’s 

i 


Examples  of  CTCs: 


1 


NASTRAN 


RADIOSS 


MADYMO 


Safety  CTCs 
with  constraints 
specified  by  . 
FMVSS  "S 
(Federal  Motor 
Vehicle  Safety 
Standards) 


y1  =  weight  of  vehicle 
y2  =  cost  of  vehicle 
^  y3  =  frontal  head  impact 


Integrated  processes  with  high  fidelity 
CAE  analyses  on  HPC  servers 


Examples  of  Critical  Design  Parameters  (CDPs  or  Xs): 


> 


v_ 


RADIOSS 

DYNA 

MADYMO 


J 


no  federal 
requirements  J 
on  these  CTCs 


y 4  =  frontal  chest  impact 
y5  =  toe  board  intrusion 
y6  =  hip  deflection 
y7  =  rollover  impact 
y8  =  side  impact 

/"  y9  =  internal  aerodynamics  (airflow) 
y10  =  external  aerodynamics  (airflow) 

=  noise 

y12  =  vibration  (e.g.,  steering  wheel) 
y13  =  harshness  (e.g.,  over  bumps,  shocks) 
^  y14  =  durability  (at  100K  miles)  > 


CFD 


x1  =  roof  panel  material 
x2  =  roof  panel  thickness 
x3  =  door  pillar  dimensions  =^>  i  beam 
x4  =  shape/geometry 
x5  =  windshield  glass 
x6  =  hood  material,  sizing  and  thickness 
x7  =  under  hood  panel  material,  sizing  and  thickness 


ti 

* 

*2 

<- 

>  NASTRAN 
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Simplify,  Perfect,  Innovate 


Introduction  to  High  Throughput  Testing  (HTT) 


•  A  recently  developed  technique  based  on  combinatorics 

•  Used  to  test  myriad  combinations  of  many  factors  (typically  qualitative) 
where  the  factors  could  have  many  levels 

•  Uses  a  minimum  number  of  runs  or  combinations  to  do  this 


Software  (e.g.,  ProTest)  is  needed  to  select  the  minimal  subset  of  all 
possible  combinations  to  be  tested  so  that  all  n-way  combinations  are  tested. 

HTT  is  not  a  DOE  technique,  although  the  terminology  is  similar 


1 

Vi 


A,* 

Academy 

A  . 


•  A  run  or  row  in  an  HTT  matrix  is,  like  DOE,  a  combination  of  different  factor 
levels  which,  after  being  tested,  will  result  in  a  successful  or  failed  run 

•  HTT  has  its  origins  in  the  pharmaceutical  business  where  in  drug  discovery 
many  chemical  compounds  are  combined  together  (combinatorial  chemistry) 
at  many  different  strengths  to  try  to  produce  a  reaction. 

•  Other  industries  are  now  using  HTT,  e.g.,  software  testing,  materials 
discovery,  interoperability  testing,  IT  (see  IT  example  on  next  page) 
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Simplify,  Perfect,  Innovate 


HTT  Example 


An  IT  function  in  a  company  wanted  to  test  all  2-way  combinations  of  a 
variety  of  computer  configuration-related  options  or  levels  to  see  if  they 
would  function  properly  together. 

Here  are  the  factors  with  each  of  their  options: 


Motherboards  (5) 
RAM  (3) 

BIOS  (3) 

CD  (3) 

Monitor  (5) 
Printer  (3) 
Voltage  (2) 
Resolution  (2) 


Gateway,  ASUS,  Micronics,  Dell,  Compaq 
128  MB,  256  MB,  512  MB 
Dell,  Award,  Generic 
Generic,  Teac,  Sony 
ViewSonic,  Sony,  KDS,  NEC,  Generic 
HP,  Lexmark,  Cannon 
220,  110 

800x600,  1024x768 


How  many  total  combinations  are  there? 


What  is  the  minimum  number  of  these  combinations  we  will  have  to  test 
(and  which  ones  are  they)  in  order  to  determine  if  every  2-way  combination 
(e.g.,  Dell  Bios  with  Teac  CD)  will  indeed  work  properly  together? 


•  To  answer  this  question,  we  used  Pro-Test  software.  The  answer  is  25 
runs  and  those  25  combinations  are  shown  on  the  next  page. 
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Simplify,  Perfect,  Innovate 


High  Throughput  Testing  (HTT) 

(for  all  two-way  combinations) 


5  Levels 

3  Levels 

3  Levels 

3  Levels 

5  Levels 

3  Levels 

2  Levels 

2  Levels 

Motherboard 

RAM 

BIOS 

CD 

Monitor 

Printer 

Voltage 

Resolution 

Case  1 

ASUS 

256  MB 

Dell 

Generic 

ViewSonic 

Lexmark 

110  V 

800  x  600 

Case  2 

Compaq 

512  MB 

Dell 

Teac 

Sony 

HP 

220  V 

1024x768 

Case  3 

Gateway 

128  MB 

Generic 

Sony 

KDS 

Cannon 

220  V 

800  x  600 

Case  4 

Dell 

128  MB 

Award 

Teac 

NEC 

Cannon 

110  V 

1024x768 

Case  5 

Micronics 

256  MB 

Generic 

Teac 

Generic 

Lexmark 

220  V 

1024x768 

Case  6 

Gateway 

256  MB 

Award 

Sony 

Sony 

HP 

110  V 

1024x768 

Case  7 

Micronics 

512  MB 

Award 

Generic 

ViewSonic 

Cannon 

220  V 

1024x768 

Case  8 

ASUS 

512  MB 

Generic 

Teac 

KDS 

HP 

220  V 

1024x768 

Case  9 

Compaq 

128  MB 

Award 

Generic 

Generic 

HP 

110  V 

800  x  600 

Case  1 0 

Micronics 

512  MB 

Generic 

Teac 

Sony 

Lexmark 

110  V 

800  x  600 

Case  1 1 

Dell 

256  MB 

Award 

Generic 

KDS 

Lexmark 

110  V 

1024x768 

Case  12 

Gateway 

512  MB 

Dell 

Sony 

Generic 

Lexmark 

110  V 

1024x768 

Case  1 3 

Compaq 

256  MB 

Generic 

Sony 

ViewSonic 

Cannon 

220  V 

1024x768 

Case  14 

ASUS 

128  MB 

Dell 

Sony 

NEC 

Cannon 

220  V 

800  x  600 

Case  1 5 

Micronics 

128  MB 

Dell 

Sony 

KDS 

Lexmark 

220  V 

800  x  600 

Case  1 6 

Gateway 

128  MB 

Generic 

Teac 

ViewSonic 

HP 

110  V 

800  x  600 

Case  1 7 

Dell 

128  MB 

Dell 

Sony 

Sony 

Cannon 

110  V 

1024x768 

Case  1 8 

ASUS 

256  MB 

Award 

Sony 

Generic 

Cannon 

220  V 

1024x768 

Case  1 9 

Compaq 

512  MB 

Dell 

Sony 

NEC 

Lexmark 

110  V 

800  x  600 

Case  20 

Gateway 

256  MB 

Generic 

Generic 

NEC 

Cannon 

220  V 

800  x  600 

Case  21 

Micronics 

512  MB 

Generic 

Teac 

NEC 

HP 

220  V 

800  x  600 

Case  22 

ASUS 

256  MB 

Generic 

Generic 

Sony 

HP 

110  V 

800  x  600 

Case  23 

Dell 

512  MB 

Generic 

Sony 

ViewSonic 

HP 

220  V 

1024x768 

Case  24 

Compaq 

256  MB 

Dell 

Generic 

KDS 

Cannon 

220  V 

1024x768 

Case  25 

Dell 

128  MB 

Generic 

Sony 

Generic 

HP 

110  V 

800  x  600 
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Example  of  Iterative  Approach  to  Modeling  and 
Simulation  to  Optimize  Transmission  Performance 


29  x’s 


Phase  1 


145-run 
LHSon 
Easy  5 


Red 


:  High  Fidelity 
Simulation  Model 


Quadratic 

DFSS 

25  Confirmation 

Predicted 

Selected 

Response  ^ 

Master: 

5  Potential 

Runs  V  of  5 

Cpksin  ( 

Best 

Surface  on  29 

Robust 

Designs 

Designs  on 

Presence 

Starting 

Variables 

Design 

Easy  5 

of  Noise 

Design 

Blue 


Low  Fidelity 
Analytical  Model 

LHS:  Latin  Hypercube  Sampling  -  a  method  for 
sampling  a  design  space  when  deterministic 
simulation  is  being  used  in  high  dimensions, 
involving  many  input  variables. 


Monte  Carlo  Simulation 
with  various  weightings 
of  responses 


t 


25-Level  LHS 
inserted  Noise  in 
each  of  8  Noise 
Variables  and  21 
Design  Variables 
(testing  the 
designs  in  the 
presence  of  noise) 


f 


Also,  reduced  to 
11  Design 
Variables  plus  8 
Noise  Variables 


Academy 


LHS  used  to  generate  5-rep  outer 
array  from  8  Noise  Variables 


Phase  Design  Vg 


8  Noise  Var 


211-4  CCD 
with  LHS  5- 
rep  Outer 
Array  on 
Easy  5 


f 


Quadratic  Model 
y  model 
§  model 


DOE  Pro 
Multiple 
Response 
Optimization 


5  Potential 

Designs 


25  Confirmation 
Runs  V  of  5 
Designs  on 
Easy  5 


CCD  Inner  Array 
Alters  1 1  Design 
Variables 


Nb! 


25-Level  LHS  inserted 
Noise  in  each  of  8 
Noise  Variables  and  21 
Design  Variables 
(testing  the  designs  in 
the  presence  of  noise) 


Simplify,  Perfect,  Innovate 


Used  with  permission  from  John  Deere 
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Optimal 

Design 


Build  and 
Test 

Prototype 


Technologies  Used  on  Transmission  Project 


•  High  Throughput  Testing  (HTT) 

To  generate  a  minimal  number  of  test  cases;  in  this  scenario,  1 1 
combinations  of  29  variables  that  would  allow  testing  all  two-way 
combinations  on  Easy  5.  This  made  running  the  Easy  5  simulator 
much  easier,  without  interruption. 


DFSS  Master:  for  Robust  Design  and  Expected 
Value  Analysis  (Monte  Carlo  Techniques) 


Highly  Orthogonal  Latin  Hypercube  Sampling 


To  populate  the  design  space  with  test  cases  which  are  highly 
orthogonal.  Typically  used  with  deterministic  simulation  to  screen  out 
the  CDPs  and  also  to  use  modeling  DOEs  on  the  simulator  to  generate 
transfer  functions  which  characterize  the  simulator 


t£F 

Academy 
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*  DOE  PRO:  for  Multiple  Response  Optimization 
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Simplify,  Perfect,  Innovate 


Summary  of  "Modeling  the  Simulator" 


Simplify,  Perfect,  Innovate 


Build  Prototype  to  Validate 
Design  in  Real  World 
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Simplify,  Perfect,  Innovate 


Environments  Where  Simulation  and  Modeling 

Is  Beneficial 


•  A  high  number  of  design  variables 

•  A  substantial  number  of  design  subsystems  and 
engineering  disciplines 

•  Interdependency  and  interaction  between  the  subsystems 
and  variables 

•  Multiple  response  variables 

•  Need  to  characterize  the  system  at  a  higher  level  of 
abstraction 

•  Time  and/or  space  must  be  compressed 
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Key  Take-Aways 
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Simplify,  Perfect,  Innovate 


For  More  Information,  Please  Contact 


Mark  J.  Kiemele 

Air  Academy  Associates,  LLC 

1650  Telstar  Drive,  Ste  110 
Colorado  Springs,  CO  80920 


Toll  Free:  (800)  748-1277  or  (719)  531-0777 
Facsimile:  (719)  531-0778 
Email:  aapa@airacad.com 
Website:  www.airacad.com 


Academy 


Associates 


Simplify,  Perfect,  Innovate 
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Live,  Virtual,  &  Constructive 
Simulation  Use 
for  Unmanned  Vehicle 
Requirements  and 
Test  &  Evaluation 


Gary  S  Kollmorgen,  Sr.  Business  Program  Manager 
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Bringing  It  Together 
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M&S/T&E  Background 
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INDEX 


M&S  Environment 


Bringing  It  Together 
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T&E/M&S  Context 


DOT&E  Policy  Memo  (4  J  une  2002) 

♦  ...we  must  continue  to  focus  on,  "...the  real  system,  in  the 
real  environment,  with  the  real  operator..." 

...however... 

♦  Models  should  help  us  predict  performance  throughout  the 
mission  space 

♦  Models  should  help  us  design  tests  to  maximize  our  learning 
and  optimally  apply  our  resources 

♦  Models  (stimulators)  should  help  us  replicate  the 
environment  during  test  to  realistically  stress  the  system 
under  test 

Thomas  P.  Christie 
Former  Director, 

Operational  Test  and  Evaluation 
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More  T&E/M&S  Context 


^Modeling  and  Simulation  in  T&E 

■  Modeling  and  Simulation  (M&S)  is  integral  to  and 
inseparable  from  T&E  in  support  of  acquisition.  For 
T&E,  M&S  is  an  essential  and  proven  tool.  Each 
military  department  has  extensive  guidelines  for  use 
of  M&S  in  acquisition  and  in  T&E.  These  guidelines 
are  intended  to  supplement  other  such  resources. 

-  Defense  Acquisition  Guidebook 

■  Appropriate  use  of  accredited  models  and  simulation 
shall  support  DT&E,  IOT&E,  and  LFT&E. 

-  DoD  Instruction  5000.2 
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M&S  Environment 


J 


Bringing  It  Together 
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Some  Context 


National  Defense  Authorization  Act  FY  2001 

♦  It  shall  be  a  goal  of  the  Armed  Forces  to  achieve 
the  fielding  of  unmanned,  remotely  controlled 
technology  such  that  -  bv  2010.  one- third  of  the 
operational  deep  strike  aircraft  of  the  Armed 

Forces  are  unmanned:  and  bv  2015.  one  third  of 
the  operational  around  combat  vehicles  of  the 

Armed  Forces  are  unmanned." 
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INDEX  -  Intelligent  Decision  Exploration 


Focusing  on 

♦Mission  test  &  planning  support  for 
unmanned  systems  in  ground  unit  protection 
scenarios 

♦Mission  test  &  rehearsal  for  unmanned 
systems  using  Live,  Virtual,  and  Constructive 
(LVC)  M&S 

♦Overall  T&E  of  unmanned  systems  especially 
with  LVC  M&S 


For  use  by  the  Planner  /  Warfighter 
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Video  Feeds 


3D  Terrain 
Visualization 


C2  Vehicle 


AN/PPS-5D 

GSR 


SeaFLIR 


RSS  w/Sensor  Suite 


Unmanned  Vehicles:  Ground  Side 
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Unmanned  Ground  Vehicle  Control  using 
Joint  Battlespace  Command  &  Control  System  (JBC2S) 


Lay  Out  Route  Points 


TAGS  Diagram 


PA TROL 


Camera  Mode 


tf  »- 


Pronf  Driving  Camera 


Control  &  Monitor 


TAGS  Ml 

Conlrof  «dWv.' 


Coffibs  GPS 


l tel  Battety 


S-'ii  rty,  etS 

i  tk  >  ^ 
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DM91 


JBC2S 


♦  Based  on: 

■  Mobile  Detection  Assessment  Response  System 
(MDARS) 

■  Multiple  Resource  Host  Architecture  (MRHA) 

■  Multi-Robot  Operator  Control  Unit  (MOCU) 


24th  National  Test  &  Evaluation  Conference 
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Mission  PAGES 


E:\My  Documents\_Projects\INDEX  07\MissionPages\testPlanV-4,xml  *  -  Mission  PAGES 


File  Edit 


Plan 


|tt|  P 

Plan  Elements 


□  BAIS  Asset-3 

□  BAIS  Asset-4 

□  I-  BAIS  Asset-5 

■  □  Cl  BAIS  Asset-6 

□  BAIS  Asset-7 

□  BAIS  Asset-8 

□  BAIS  Asset-9 

•ni-ibMtWBMB 

■  n  R4IS4^t-U 


Plan  Element  Properties 


::: 

ti  H  4 

-3  Misc 

VVj 

Category 

Ground 

Threat 

Friendly 

-  : 

Static 

4B. 

Min  Speed 

0.0  km  (0,0) 

Max  Speed 

0,0  km  (0.0) 

B  BAIS 

V 

Metrics 


Metric  Name 


Result 


Closest  Standoff 
Average  Standoff 


Weight 


0.0  km 


0.0  km 


Total  Metric:  0.0 


Latitude.  Longitude  (30.58765.  -86.45497) 


Mission  Planning  and  Graphical  Editing  System 
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Current  Capability 


♦Assist  Military  Planners 

■  Quickly  create  deployment  plans  for  multiple  UVs  in 
response  to  a  new  unit  protection  threat. 

■  Evaluate  unit  protection  plans  w/CONOPS-oriented 
metrics 

♦“What  if”  planning  and  evaluation 

■  Effects  of  changing  deployment  patterns 

■  Effects  of  degraded  sensors 


24th  National  Test  &  Evaluation  Conference 


referentia  13 


Contents 


M&S/T&E  Background 


INDEX 


M&S  Environment 


Bringing  It  Together 
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Notional  Architecture 


Instructor  Other 

Operator  Station  Components 
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Distributed  M&S 


♦  Robust  capabilities  exist  today 

■  Computer  Generated  Entities  (Constructive) 

-  JSAF  (Navy,  JFCOM),  JCATS  (LLNL),  OneSAF  (Army), 
VRForces  (Commercial) 

■  Users:  Army/Navy/Air  Force  (T&E,  Training), 

Joint  (Experimentation,  CONOPS,  and  Training) 

■  C4I  connectivity 

■  Use  real  world  systems  to  control  simulated  assets 

■  Joint  LVC  Data  Translator,  SIMPLE,  others 

■  Visualization 

■  Really  too  numerous  to  mention 

■  Standards  based  communication  protocols 

■  Distributed  Interactive  Simulation  (DIS) 

■  High  Level  Architecture  (HLA) 

■  Voice  over  Internet  Protocol  (VoIP) 
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Virtual  Simulators 
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“Real  World”  Command  &  Control 


+  Example:  Advanced 
Field  Artillery 
Tactical  Data 
System  (AFATDS) 

■  Processes  fire  mission 
and  related 
information  to 
coordinate/optimize 
the  use  of  all  fire 
support  assets 

■  Includes  mortars,  field 
artillery,  cannon, 
missile,  attack 
helicopters,  air 
support,  and  naval 
gunfire 
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C4I  Connectivity 
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Urban  Environment  -  3D  Viewers 
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Synthetic  Natural  Environment 


Cloud  Cover,  Sun  Angle,  Night  &  Day  Obscurants:  Snow,  Rain,  Dust 


Weapon  fly  outs,  contrails  Dynamic  changes:  Debris,  Fire,  Smoke 
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Urban  Environment  &  Culture 
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Testing  Control  Station 


fm. 

4  Bum. 

SSL 

fiT 
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*4 
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♦  For  LVC  environments 

■  Planning 

■  V/C  scenario  generation 

■  V/C  scenario 
execution/exercise  control 

■  After  Action 
Review/Results 

■  Test  environment  control 

♦  Framework  is 
configurable  for 
specific  testing 
environments. 

♦  Support  for  distributed 
and  collaborative 
operations. 
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After  Action  Review  (Test  Data) 


M  Timeline  for  vehicle  20G24  (  LI  300  TEL  ) 


Timeline  Resolution 


00:30 


01:00 


01:30 


02:00 


02:30 


03:00 


03:30 


LEAVE  LAUNCH  AREA  -> 


®K-Kill[  02:33:55  ] 
STOP_MOVE  ->  START_MOVE[  02:04:37  ] 
#CAMQ  Change:  CAMO_4  ->  CAIvIO_2[  02:04: 

LEAVE  RESUPPLY  ->  ENTER 


♦Future  TBM  (SWA)[  01:03:17  ] 

STOP_MOVE  ->  START_MOVE[  00:25:45  ] 

START_MOVE  ->  ST0P_M0VE[  00:26:39  ] 

=|  ENTER_LAUNCH_AREA[  00:26:21]  LEAVE_RESUPPLY  -  >  ENTER_RE! 

4  S  TART_MOVE  -  >  S  T  0P_M0 VE[  01:12:11]  LEAVE_LAUNCH_AREA  -  >  ENTER_LAUN 
|#C AMO  Change:  C AMO_2  -  >  C AMO_  1  [  0  0 : 3 1 : 5 1  ]  ENTER_LAUNCH_AREA  -  >  LEAVE J^AUN 

S  T  OP_MOVE  -  >  S  TART_MOVE[  01:05:44  ]  )S  TART_MOVE  -  >  S  T  OP_MO VE[ 

ENTER_LAUNCH_AREA  ->  LEAVE_LAUNCH_AREA[  01:06:01  ] 

•  C  AMO  Change:  C  AMO_  1  -  >  C  AMO_4  [01:12:14]  ST  OP_MO  VE  -  >  S  TART_MOVE[ 


START 


MOVE  ->  STOP MOVE[ 


4] 

SUPPLY[  02:18:51  ] 
!H_AREA[  02:07:42  ] 
CH_AREA[  02:08:14  ] 
02:18:55  ] 

02:18:56  ] 

02:19:04] 


W- 

Air 


16] 

Vehicle  (24839) 


NOMMedium  Pnonty .  Track  053  07  [01:14:30]  AUT  O  OVERRIDE:  Long_Range_Launcher  Track:  0  5  3  07  [  0  2 : 

PLAN:  Very  Low  Priority.  ToT:  0  Track  053 07 [  01:16:55]  ♦Attacked:  Attacker:  UCAV  Unmanned  Combat 

NOM:Low  Priority.  Track  05307[  02:17:51  ] 

♦PAIR:  Attacker:  UCAV  Unmanned  Combat  Air  Vehicle  (6669)  Munition:  FLAW  Future  Low  cost  Attack  Weapon  Pod  Track  05307[  0: 

♦PLAN:  Very  Low  Priority.  ToT:  5383602  Track  05307[  02:18:36  ] 

Executing:  Attacker:  UCAV  Unmanned  Combat  Air  Vehicle  (6669)  Munition:  FLAW  Future  Low  cost  Attack  Weapon  Pod  Track  0530 

FAIR:  Attacker:  UCAV  Unmanned  Combat  Air  Vehicle  (24839)  Munition:  FSQi 
♦Attacked:  Attacker:  Unknown  Munition:  FLAW  Future  Low  cost  Attack  Weapon  Pod  Track  053 07 [  01:28:31  ] 

♦Attacked:  Attacker:  Unknown  Munition:  FSOW-C  Future  Standof 
Executing:  Attacker:  UCAV  Unmanned  Combat  Air  Vehicle  (24839)  Munition:  I 


3  |  Unsigned  Java  Applet  Window 
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M&S/T&E  Background 


INDEX 


M&S  Environment 


Bringing  It  Together 
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CONOPs  Simulation 


^Animation  Viewer:  top.Enterprise  Network  packet  flow  /  node  movement 


JnJji! 


I  1 

In-Flight 

O 

Surface 

1  I 

Subsurface 


File  Edit  Controls  Windows  Help 
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OCX  W  [SHX 


Modeled 

Entities 

USV 

UAV 

MIW 

9 

1 

ASW 

4 

0  , 

SUW 

3 

8 

i 

Current  Frame :  0 


Time  Elasped  :  0.00  sec 


Number  of  Colors  :  32  bits 


I  ri.-i.-..-,  1 


MODEL  RESULTS 

Required 
M  Bandwidth  41 .5  MB 


CONOPs  Validated 


OOd  OOh  02m  08s  .  000ms  OOOus  000ns  OOOps 
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The  Exploration  &  Test  Environment 


JBC2S  positioned 
by  C2  Vehicle 


Data  Logging  & 
Results 


Simulated  battlespace 
is  geotypical  or 
geospecific  terrain  / 


Mission  Pages 


“Planning 


Event  Control 
2D  Viewer 


C2  Vehicle 


Visualization 


Unmanned 
Vehicles  can  be 
Live  or  simulated 


JBC2S  talks  to 
simulated  vehicles 
via  C2  interface 
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Scaleability 


UV  SENTRY  OPERATIONAL  CONCEPT 
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FORCEnet  enables  the 
Common  Operating  Picture 


FORCEnet 
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Questions? 


gkollmorgen@referentia.com 


(757)  620-1777  cell 
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Capability  Test 
Methodology  (CTM) 
Measures  Framework 


February  27,  2008 

Mr.  Max  Lorenzo 
Technical  Director 
757,638.6079 
maxJorenzo@ite.osd.nnil 


Overview 


•  TiJE 

•  JTEM  Overview 

•  Why  Measures  Framework  ? 

•  CJI  Construct 

•  CEM 

•  Measure  Framework 
-  Examples 

•  Way  Ahead 


item@ite.osd.mil 
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Why  Test  in  a  Joint  Environment? 

It  is  a  requirement . .  . 


Hey,  this  stuff  passed  its 
interoperability  certs! 
How  come  it  doesn’t 
work  in  the  AOR? 


HflEwwfcad  Fim 


Single  Service  Operational  Test 
in  a  Realistic  Environment 


“OT&E  shall  determine  the 
operational  effectiveness  and 
suitability  of  a  system  under 
realistic  operational 
conditions,  including  combat.” J 


DoDI  5000.2,  E.7 
[May  2003] 


Army 

Forces 


Air 

Forces 


Deconflicted 


ENABLING  THE 
JOINT  FORCE 


Test  in  a  Joint  Environment  across  the 
Acquisition  Life  Cycle 


Interoperability 

Test/Certificatior 


“Systems  that  provide 
capabilities  for  joint 
missions  shall  be  tested  in 
the  expected  joint 
operational  environment . 


. .  any  resultant  materiel 
solution  will  be  verified 
through  testing  in  the 
expected  joint  operational 
environment  to  demonstrate 
joint  interoperability . . 


DoDI  5000.2,  E.5 
[Draft,  August  2007] 


rjOINT  CAPABLE  FORCES 


CJCSI  3170.01  F,  B.3 
[May  2007] 


Joint 


Interdependent 


Integrated 


Coordinated 


It  is  the  right  thing  to  do  .  .  . 

-  Early  discovery  of  problems,  reduced  rework  costs 

-  Improved  test  data  for  milestone  decision  authorities 


Improved  system  characterization  and  limitations  for  Service  and 
combatant  commander  planning 


-  Field  proven  joint  capabilities  to  the  combatant  commander 
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Testing  in  a  Joint  Environment  (TiJE) 
Methods  and  Processes  Approach 


Capability  Test 
Methodology  (CTM) 


Capability 

Evaluation 

Metamodel 

(CEM) 


Joint  Mission 
Environment 
Foundation  Model 
(JFM) 


Process  Anomaly 
Reports  (PAR) 


•  Best  practices 

•  Consistent  approach  to  joint 
mission  environment  (JME) 

•  Consistent  joint  capability 
assessments  and  evaluations 

^Application  across  the 
acquisition  life  cycle 
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Why  Measures  Framework? 


•  For  consistent  answers  on  SoS,  need 
standard  methodologies  across: 

-  Services 

-  Domains 

•  Need  standard  instrumentation 

•  Need  joint  tasks  measured  by  test 
organizations 

•  Need  measures  rooted  in  JCIDS,  AA,  etc. 
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Relationship  between  CEM  and  CTM 


Capability  Test  Methodology 
(CTM)  Lexicon 


provides  underlying 
conceptual  definitions 


Capability  Evaluation 
Metamodel  (CEM) 


provides  underlying 
business  rule 
relationships 


LVC-DE  -  Live,  Virtual,  Constructive  Distributed  Environment 
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Capability  Evaluation  Metamodel 

(CEM) 

•  JTEM  is  designing  a  Capability  Evaluation  Metamodel 
(CEM) 

-  Metamodel  definition 

•  An  explicit  model  of  the  constructs  and  rules  needed  to  build 
specific  models  within  a  domain  of  interest 

•  A  meta-model  can  be  viewed  from  three  different  perspectives: 

-  as  a  set  of  building  blocks  and  rules  used  to  build  models 

-  as  a  model  of  a  domain  of  interest,  and 

-  as  an  instance  of  another  model 

-  CEM  purpose 

•  Provide  an  underlying  structure  for  CTM  evaluation  methods 
and  processes  (for  example,  CTM  evaluation-related  template 
content  and  relationships) 

•  Provide  business  rule  relationships  for  the  CTM  evaluation 
thread 
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Analytic 

Agenda 


Provide  COCOM 


JV 


80 
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CEM  Key  Relationships 


Joint  Mission  Desired  Effects 

’  Disrupt  Red  Combat  forces  moving  in  battle  area. 

•  Attrite  Red  Combat  forces  in  the  battle  area.  i 

•  Support  Blue  maneuver  in  the  battle  area.  Achieve 

•  Protect  Blue  forces  in  the  battle  area. 


Mission  MOEs 


Joint  Capability 

Integrated  Fires  System  of  Systems 

Recon/Surveillanci 

Airborne  C2  *< - >-  GPS  | i  * 

It.  mi 


Joint 

Tasks 

SN  1,  “Conduct 
Strategic 
Deployment  and 
Redeployment” 

OP  1,  “Conduct 
Operational 
Movement  and 
Maneuver” 

TA  1, 

“Deploy/Conduct 

Maneuver” 


Task  MOPs 


UJTL 


System  of  Systems 


SoS  Attributes 


CTM  establishes  the  M&P  to  test  SoS  ability  to  provide  the  means  and  ways  to  perform  a  set  of  tasks 
in  order  to  achieve  the  set  of  desired  effects  that  lead  to  mission  success 


CEM  -  Capability  Evaluation  Metamodel 

JCIDS  -  Joint  Capabilities  Integration  and  Development  System 

MOP  -  Measure  of  Performance 


CTM  -  Capability  Test  Methodology 
M&P  -  Methods  and  Processes 
SoS  -  System  of  Systems 


JCA  -  Joint  Capability  Area 
MOE  -  Measure  of  Effectiveness 
UJTL  -  Universal  Joint  Task  List 
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Capability  Evaluation  Metamodel  (CEM): 

Test  Plan  Design 


MISSION  DESIRED 
EFFECT 

intended 
to  achieve 

i 

ESSENTIAL 
TASK 

"S' 


BLUE  SYSTEM  OF 
SYSTEMS 
(across  DOTMLPF) 


references 
capability  ■ 
crosswalk 
elements 
concerning 


CRITICAL 
JOINT  ISSUE 
(CJI) 


TEST  GOAL 
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task 

performance" 
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PERFORMA 
(MOP) 
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NCE 

- J 

i 


states 
mission 
effect, 
task,  SoS 
attribute, 
condition 
dependent  elements 
variable 
measure 
of 


states  a 
capability 
subset 
focus  for 


is 


SYSTEM/SoS 

is  \ 

ATTRIBUTE 

dependent  * 

_ _ ■ 

variable 

measure  . 

of  V 

provides 

specific 

CJI 

objective 

for 


specifies 

joint 

operational 

descriptions 

for 


TEST  TRIAL 


7 


1 


THREAT 
SYSTEM  OF 
SYSTEMS 


ENVIRONMENTAL 

CONDITION 


IS 

independent* 

■  variable 
factor  of 

is 

independent' 
variable 
factor  of 


CAPABILITY 

focuses  on 

CAPABILITY 

TEST 

^  specific 

SUBSET  TEST 

DESIGN 

CJIs  using 

PLAN  DESIGN 

_  ^ 

is  specific 
factor,  level, 
and 

measure 
combination 
of 


compares 

actual 

responses” 

to 


item@ite.osd.mil 


Legend 


Joint  Operational 
Context  for  Test 
(JOC-T) 


CEM  Measures 
Framework 


CEM  Evaluation 
Concepts 


Capability  Evaluation  Metamodel  (CEM): 
DOD  Context  for  CEM  JOC-T 
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Global 


Metamodel  (CEM): 
View 
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Evaluation  Strategy 
Overview 


•  Critical  Joint  Issue  (CJI)  Checklist 

-  Is  the  CJI: 

•  Used  to  assess  performance  as  it  pertains  to  capabilities  supporting  joint 
missions? 

•  Structured  as  a  question  addressing  joint  capability  areas  described  in  Joint 
Capability  Documentation? 

•  Addressing  the  system  of  systems  ability  to  perform  joint  operational  tasks 
and/or  the  system  of  systems,  system,  or  service  attribute  performance? 

-  Does  the  CJI  phrasing  include  SoS  contribution  to  achieving  the 
desired  mission  end  state  outcomes  in  terms  of  mission  desired 
effects? 

-  CJIs  should  be  of  primary  importance  to  the  decision  authority  in 
reaching  a  decision  to  allow  the  system  of  systems  to  advance  into  the 
next  phase  of  development. 

•  CJI  Example  Template: 

Can  the  Capability  perform  Task  X  by  SoS  Configuration  Y  under 
Condition  A  to  achieve  Mission  Desired  Effect  Z? 
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Measures  Framework 


Service-acquired 

capabilities 


that  perform 
joint  tasks 


\ 


CJI:  Can  JBD2  perform  to  enable  close  air  support  by  near- 
term  networked  fires  SoS  under  a  hostile  threat  environment 
to  achieve  destroyed  or  neutralized  Threat  Forces  (in  Zone, 
^  moving  into  Zone,  indirect  fires,  ADA)? 


T 


to  accomplish  a 
joint  mission 


Critical  joint  issues  assess  system  and  task  performance  pertaining 
to  capabilities  which  support  joint  missions. 


item@ite.osd.mil 


13 


Measures  Framework 


COIs  & 
system/SoS 
attributes  (KPP, 
CTPs) 


CJI:  Can  j7*D2  perform  to  enable  close  air  support  by  near- 
term  networked  fires  SoS  under  a  hostile  threat  environment 
to  achieve  destroyed  or  neutralized  Threat  Forces  (in  Zone, 
moving  into  Zone,  indirect  fires,  ADA)? 


•  Systems  measure  critical  technical  parameters  (CTP) 
(attributes,  that  when  achieved,  allow  attainment  of 
desired  operational  capabilities)  to  resolve  critical 
operational  issues  (COI)  regarding  task  accomplishment 


The  program  manager/OTA  need  to  know  if  the  SUT  can  accomplish 
the  required  task  in  the  Joint  Operational  Context  for  Test. 
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Measures  Framework 


•  Systems  are  combined  to  accomplish  Universal 
Joint  Tasks  which  are  measured  at  the  System  of 
Systems  level  (Services  acting  in  concert  to  achieve 
a  task) 


Joint  tasks  are  measured  by  a  Task  Measure  of  Performance  derived 
from  UJTL  references  (after  Joint  Capability  Area  analysis). 
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Measures  Framework 


Mission  Measure  of  Effectiveness  (MMOE)  measures  desired  end 
state(s)  (desired  effects)  that  result  from  task  performance. 
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Integrated  Fires  Capability  Operational  Concept  Graphic  (OV-1) 


Mission  Statement 

Blue  Forces  conduct  Joint  Forcible 
Entry  Operations  to  expand  lodgment 
and  control  key  infrastructure  in  order 
to  facilitate  rapid  force  build-up  in  the 
Joint  Operations  Area  (JOA) 


End  State  Desired  Effects 

•  Expand  lodgment  and  control  key  •  Destroyed  or  neutralized  Threat  Forces  (in 

infrastructure  to  facilitate  rapid  force  build-up  Zone,  moving  into  Zone,  indirect  fires,  ADA) 

•  Preserve  key  infrastructure  and  prevent  •  Blue  unhindered  in  maneuver  within  the  JOA 

environmental  contamination  •  Blue  force  survivability  in  the  JOA 

•  Secure  LOCs  to  facilitate  flow  of  forces 

needed  to  conduct  regime  change  operations_ y1 


Integrated  Fires  System  of  Systems 


i  gfe 
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rVf  iRp 


USMC 


*  » 


Networked  Fires 
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Tasks 
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Mission  MOE:  Threat  Systems  Combat  Ineffectiveness 

(Percentage  of  threat  systems  rendered  ineffective 
[killed,  damaged,  dislocated]  compared  to  Blue  desired  value  ) 


Mission  Statement 

Blue  forces  conduct  joint  forcible  entry 
operations  to  expand  lodgment  and  control 
key  infrastructure  in  order  to  facilitate  rapid 
force  build-up  in  the  joint  operations  area 
(JOA) 

End  State 

Expand  lodgment  and  control  key 
infrastructure  to  facilitate  rapid  force  build-up 

Desired  Mission  Effect 

Destroyed  or  neutralized  threat  forces  (in 
zone,  moving  into  zone,  indirect  fires,  ADA) 


©CD 

to 

Trial  Start 


am 

DCGS-A 


~7~. /.  i' -l _ V '  \ 


Networked  Fires 
Air  Maneuver/Operations 
Air  and  Missile  Defense 


(D(D 


(D 


(D 


dXD 


Joint  Mission  Time  Horizon 


Trial  End 


Sequence  ofEvents 


1 .  Mission  start  at  time  t0.  Threat  forces  in  JOA  or  moving  into  JOA.  Blue  forces  conducting  joint  forcible  entry  operations. 

2.  Blue  forces  employ  Networked  Fires  task  thread  to  prosecute  threat  targets. 

3.  Threat  systems  killed,  damaged,  or  dislocated. 

4.  Mission  complete  at  time  tn 
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Mission  MOE:  Threat  Systems  Combat  Ineffectiveness 

(Percentage  of  threat  systems  rendered  ineffective 
[killed,  damaged,  dislocated]  compared  to  Blue  desired  value  ) 


Data  Elements 


1.  Blue  Desired  Fraction  of  Threat  Systems  in  JOA  Combat  Ineffective  at  Mission  End  State  (threshold  value) 

2.  Cumulative  count  Threat  Systems  in  zone  overtime  (System  Type,  Time,  Location) 

3.  Cumulative  count  Threat  Systems  Killed  (System  Type,  Time,  Location) 

3.  Cumulative  count  Threat  Systems  Combat  Damaged  (System  Type,  Damage  Type,  Time,  Location) 

3.  Cumulative  count  Threat  Systems  Dislocated  (System  Type,  Dislocation  Type,  Time,  Location) 


Key  Terms 


1 .  Threat  Mounted  Systems:  Those  threat  mounted  systems  that  either  are  in  the  JOA  at  trial  start  or  enter  the  JOA  during  the  trial  period.  System  does  not  have  to  still 
be  in  JOA  at  trial  end  to  be  counted.  A  system  will  also  not  be  counted  more  than  once  (if  enters  JOA  more  than  once).  (System  Type:  identifying  type;  Time:  Time  at 
which  entered  JOA;  Location:  Positioning  data  at  time  which  entered  the  JOA.  If  in  JOA  at  trial  start,  then  position  at  trial  start  time) 

2.  Threat  Mounted  System  kill:  Those  threat  systems  that  can  no  longer  move,  shoot,  and  sense.  System  no  longer  possesses  a  capability.  Assumes  cannot  be  fixed 
and  remains  a  kill  until  trial  end.  (System  Type:  identifying  type;  Time:  Time  at  which  killed;  Location:  Positioning  data  at  time  which  killed) 

3.  Threat  Mounted  System  damaged:  Those  threat  systems  that  can  no  longer  shoot  but  are  not  killed.  System  is  combat  ineffective  but  still  retains  some  capability  to 
move  or  sense.  (System  Type:  identifying  type;  Damage  type:  S  -  unable  to  shoot,  SM  -  unable  to  shoot  and  move,  SS  -  unable  to  shoot  and  sense;  Time:  Time  at 
which  entered  damaged  state.  Must  remain  in  damaged  state  to  trial  end  to  be  counted.  If  later  attacked  and  killed,  then  counted  as  a  kill.  If  moves  out  of  JOA  or  goes  into 
hiding,  still  considered  damaged  and  not  dislocate;  Location:  Positioning  data  at  time  which  entered  damaged  state) 

4.  Threat  Mounted  System  dislocated:  Those  threat  systems  that  are  not  combat  effective  in  the  JOA  at  trial  end.  System  either  leaves  the  JOA  and  remains  outside  the 
JOA  until  trail  end,  or  system  is  in  the  JOA  and  hiding  at  trial  end  to  avoid  attack.  Hiding  assumes  it  is  unable  to  move,  shoot,  or  sense.  A  system  is  only  counted  as 
dislocated  if  in  that  state  at  trial  end.  (System  Type:  identifying  type;  Dislocation  type:  departed,  denied,  hidden;  Time:  Time  at  which  entered  dislocated  state.  Reset  if 
system  goes  out  of  dislocated  state  before  trial  end;  Location:  Positioning  data  at  time  which  entered  dislocated  state.  Assumes  does  not  move  if  remains  in  dislocated 
state  until  trial  end) 


Calculation 


r  X  Threat  Systems  Combat  Ineffective  (killed  v  damaged  v  dislocated)  in  JOA  at  trial  end 
v  X  Threat  Systems  in/newly  entered  JOA  by  trial  end  y 

(Blue  Desired  Fraction  of  Threat  Systems  Combat  Ineffective  in  JOA  at  Mission  End  State) 


Success  Criteria 


MMOE>  1.0 
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Task  MOP:  Percentage  targets  attacked  IAW  requests  for  fires 
(Number  of  targets  attacked  compared  to  total  number  of  requests  for  fires) 


NLOS-LS 


JTF  SOF 


152mm  Self-Propelled 
Howitzer  Battery  (2S19) 


Mission  Statement 

Blue  Forces  conduct  Joint  Forcible 
Entry  Operations  to  expand  lodgment 
and  control  key  infrastructure  in  order 
to  facilitate  rapid  force  build-up  in  the 
Joint  Operations  Area  (JOA) 

Joint  Task 

TA  3.2. 1  Conduct  Joint  Fires: 
Employ  Fires  that  support  land, 
maritime,  amphibious,  and  special 
operation  forces  to  engage  enemy 
forces,  combat  formations,  and 
facilities  in  pursuit  of  tactical  and 
operational  objectives. 


MOP  (Ml) 

Percentage  targets  attacked  IAW 
requests  for  fires 


Task  Start 


Joint  Task  Time  Horizon 


©  ® 


tf 

Task  End 


■*tn 


Trial  End 


Seguence  ofEvents 

1.  JTF  SOF  soldier  detects  2S19  Battery  at  start  time  td 

2.  JTF  SOF  soldier  reports  2S19  target  information  to  the  FEC  to  populate  the  COP. 

3.  Decision  is  made  to  engage  the  2S19s  with  PAM.  Request  for  fires  is  made. 

4.  NLOS-LS  receives  fire  mission 

5.  NLOS-LS  processes  the  target  acquisition  information  and  is  prepared  to  fire  PAM. 

6.  NLOS-LS  fires  PAM  at  time  tf 

7.  Class  III  UAV  detects  effects. 

8.  BDA  report  is  sent  to  FEC  for  COP  update  and  potential  re-targeting/re-attack. 

9.  Ground  truth  of  target  effects. 


■O 
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Task  MOP:  Percentage  targets  attacked  IAW  requests  for  fires 

(Number  of  targets  attacked  compared  to  total  number  of  requests  for  fires) 


Data  Elements 


1.  Blue  Target  Engagement  Objective  Value 
3.  Number  of  requests  for  fire 

6.  Number  threat  systems  attacked  from  requests  for  fire 


I - 1 


Key  Terms 

1 .  Threat  Systems  (TS^)  detected  in  JOA :  Those  threat  systems  (TSd)  that  are  detected  in  the  JOA  and  a  request  for  fires  is  executed.  (System  Type:  identifying  type; 
Detection  time:  Time  at  which  TS  was  first  detected  (td.),  Request  for  Fires  time:  Time  at  which  request  for  fires  on  TS  was  executed  (tr.) 

6.  Threat  Systems  (TSJ  targeted  in  JOA  :  Those  threat  systems  (TSk)  that  are  targeted  and  attacked  in  the  JOA  during  the  trial  period.  (System  Type:  identifying  type; 
Fires  time:  Time  at  which  TMP  was  fired  on  (tf.) 


TMOP  = 


Calculation 

X  Threat  Systems  Attacked  from  Requests  for  Fire 
X  Requests  for  Fire 

BlueT  argetEngagementObj  ective  V  alue 


Success  Criteria 


TMOP  >  1.0 
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SoS  Attribute:  Rapid  Fires  Engagement 
(Average  time  to  engage  targets  from  networked  fires) 


COP 


SoS  Performance  Areas 

Doctrine 

Organization 

Training 

Materiel 

Leadership 

Personnel 


Mission  Statement 

Blue  Forces  conduct  joint  forcible  entry 
operations  to  expand  lodgment  and 
control  key  infrastructure  in  order  to 
facilitate  rapid  force  build-up  in  the 
joint  operations  area  (JOA) 

System  of  Systems  (SoS) 

Networked  Fires 


NLOS-LS 


JTF  SOF 


152mm  Self-Propelled 
Howitzer  Battery  (2S19) 


Trial  Start 


-I- 


©  ® 


♦tn 


Joint  Task  Time  Horizon 


Trial  End 


Seguence  ofEvents 

1.  JTF  SOF  soldier  detects  2S19  Battery  at  start  time  td 

2.  JTF  SOF  soldier  reports  2S19  target  information  to  the  FEC  to  populate  the  COP. 

3.  Decision  is  made  to  engage  the  2S19s  with  PAM 

4.  NLOS-LS  receives  fire  mission 

5.  NLOS-LS  processes  the  target  acquisition  information  and  is  prepared  to  fire  PAM. 

6.  NLOS-LS  fires  PAM  at  time  tf 

7.  Class  III  UAV  detects  effects. 

8.  BDA  report  is  sent  to  FEC  for  COP  update  and  potential  re-targeting/re-attack. 

9.  Ground  truth  of  target  effects. 


■O 
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SoS  Attribute:  Rapid  Fires  Engagement 

(Average  time  to  engage  targets  from  networked  fires) 


Data  Elements 


1.  BlueDesiredEngagementTime 
1.  Cumulative  Count  Threat  Systems  Engaged 
1.  Threat  System  detection  time  (td) 

6.  Threat  System  fires  time  (tf) 


I - 1 


Key  Terms 

1 .  Threat  Systems  (TS^)  detected  in  JOA :  Those  threat  systems  (TSd)  that  are  detected  in  the  JOA  and  a  request  for  fires  is  executed.  (System  Type:  identifying  type; 
Detection  time:  Time  at  which  TS  was  first  detected  (td.),  Request  for  Fires  time:  Time  at  which  request  for  fires  on  TS  was  executed  (tr.) 

6.  Threat  Systems  (TSJ  targeted  in  JOA  :  Those  threat  systems  (TSk)  that  are  targeted  and  attacked  in  the  JOA  during  the  trial  period.  (System  Type:  identifying  type; 
Fires  time:  Time  at  which  TMP  was  fired  on  (tf.) 


Calculation 


BET  = 


X  (Threat SystemFiresTime  t(  -  ThreatSystemDetectionTime  td  ) 
CumulativeCountThreatSystemsEngaged 
BlueDesiredEngagementTime 


V  TSk  and  Engaged 


Success  Criteria 


BET  >  1.0 
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Path  Forward 


•  Need  Analysis  Framework 

•  Need  regular  tests  in  JCAs 

•  Need  CONUS  virtual  range  stood  up 
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Joint  Test  Director 
757.638.6099 

eileen.biorkman@8te.osd.mil 


Mr.  Max  Lorenzo 
Technical  Director 
757.638.6079 
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Test  Pilot’s  Role  in  Flight 
Test  of  Unmanned  Air 

Vehicles 

Roy  Martin 
Chief  Test  Pilot 

Northrop  Grumman  Air  Combat  Systems 


Abstract 

With  Increasing  Emphasis  on  the  Development  of 
Unmanned  Air  Vehicles  (UAV),  the  Test  Pilot  May  Be 
Assigned  to  Perform  Various  Duties  on  the  UAV  Test 
Team.  The  Purpose  of  This  Briefing  Is  to  Highlight 
Some  of  the  Tasks  That  May  Be  Assigned  to  a  Test 
Pilot  on  a  UAV  Test  Team.  Information  on 
Accomplishing  These  Tasks  from  Previous  UAV 
Programs  Are  Presented.  The  Information  Is  Derived 
Primarily  from  the  Author’s  Experience  Performing 
Duties  as  Chase  Pilot,  Surrogate  Aircraft  Test  Pilot, 
and  Air  Vehicle  Operator  on  Various  UAV  Flight  Test 
Programs. 


AVO  role  for  UAV  Analogous  to 
Test  P  lot  Role 


■  AVO  Role  Defined  in  Early  Design  Decisions 

■  Command  and  Control  Location 

■  AVO  Display  Requirements 

■  Surrogate  Aircraft  Flight  Testing 

■  Chase  Aircraft 

■  Contingency  Planning 

■  Summary  /  Lessons  Learned 


AVO  Role  Defined  in  Early 
UAV  Design  Decisions 


■  Fully  Autonomous  UAV  -  Range  Safety  Role 

■  Autonomous  Takeoff  /  Landing  UAV 

-AVO  Cannot  Manually  Takeoff  /  Land  the  UAV 

-AVO  Involved  in  Decisions  Related  to  Taxi  and 
Takeoff  Abort,  Contingency  Profiles,  Waveoff, 
Destruct 

-Limited  Capability  to  Control  Flight  Path  -  Manual 
Override  Airspeed  /  Altitude  /  Bank  Angle 

■  Manual  Control  of  Takeoff  /  Flight  Path  /  Landing 

-AVO  Fully  Responsible  for  Flight  Path  Control 
Through  Remote  Control  Inputs 


Command  and  Control  Locations 


■  AVO  in  Cockpit  of  Chase  Aircraft 
-Communication  Issues 
-Removed  from  Engineering  Decisions 

■  AVO  in  Remote  Ground  Based  Location 
-Same  Issues  as  Inflight  Location 

■  AVO  Co-Located  with  Engineering  Decisions 
-Visual  Response  Supplements  Net  Communication 
-Advantageous  to  Co-Locate  AVO  with  RSO 


Control  Room  Configurat  on 


■  AVO  Visual  with  Test  Director  /  Test  Conductor  / 
Engineers  to  Supplement  Net  Communication 

■  AVO  Share  Displayed  Information  with  RSO 

■  Radio  Communication  with  Ground  Crew  /  Chase 
Aircraft  /  Tower  /  Range  Control 

■  External  Camera  View  of  UAV  Highly  Desired 

■  Forward  Looking  Camera  from  UAV  Critical  for 
Landing  Decisions 


X-47A  Control  Room 


1 


X-47A  AVO  Station 


AVO  Command  Display  Requirements 

■  AVO  Command  and  Control  Panel 

-AVO  Selects  and  Executes  Commands  to  Be 
Uplinked  to  UAV  Using  Hardwired  Panel  or 
Computer  Display  (More  Flexible  /  Transportable) 

-Need  Feedback  That  Uplinked  Command  Was  Sent 
(Use  of  Color  Change  on  Display  Is  Effective) 

-“Echo”  Command  Is  Useful 

■  Use  of  “Mouse”  to  Select  Commands  on  Display 

■  Limited  Keyboard  Entry  (i.e.,  Altimeter  Setting,  ZFW, 
Go  to  WP,  Mission  Plan  Number,  etc.) 


Sample  Command  D  splay 


AVO  Systems  Monitor  Display  Requirements 


■  Vehicle  Status  Display 

-Monitor  Current  UAV  Attitude,  Flight  Conditions, 
Basic  System  Status 

-Monitor  Planned  UAV  Flight  Conditions 

-Monitor  Uplink  and  Downlink  Status 

-Feedback  That  “Commands”  Were  Received 

-Distance  to  Go  to  Waypoints  -  Important 

-Telemetry  Drop-Outs  Are  Really  Irritating 

-Consider  Backup  Method  to  TM  Downlink 
(i.e.,  Command  Data  Link  -  Limited  Bandwidth) 


Sample  Vehicle  Status  Display 


Mission  Situation  Display  (Range  Map) 


■  Presents  “Gods  Eye”  View  of  UAV  Position  in 
Relation  to  Planned  Mission  Track  /  Waypoints 

-Shows  Waypoints  /  Routes  -  Both  Normal  and 
Contingency  Mission  Plans 

-Need  “Declutter”  of  Unused  Route  Segments 

-Need  “Quick”  Transition  to  Desired  Segments 

-Center  on  UAV  With  Ability  to  Vary  Map  Size 

-Display  of  UAV  “Track  History”  Is  Very  Useful 


Sample  Mission  Situation  Display 


Surrogate  Aircraft  FI  ght  Testing 


■  Manned  Aircraft  With  “Some”  UAV  Systems  Installed 
-Check  RF  Links  (Command  and  Control  /  TM) 

-Check  RF  Links  Used  by  UAV  (GPS  /  Diff  GPS) 

-Verify  Navigation  Software 

-AVO  Range  Familiarization  Is  by  Product  of  Testing 

■  Aircraft  Selection 

-Verifying  UAV  Track  Guidance  Is  Primary  Task 

-Replicate  UAV  Altitude  and  Airspeed  Is  Optional 

-Keep  It  Simple  (Display  Range  Map  /  Route  /  UAV 
Position  on  Laptop  in  Aircraft  Reduces  Aircraft  Mods) 


Surrogate  Testing 

■  Testing  with  Beech  Barron 
Provides  the  Following: 

-Data  Link  Range  Limitations 

-Data  Link  Null  Locations 

-Profile  Link  Reception 

-Quantitative  NAV  Performance 


-Integrated  NAV /  Mission 
Plan  /  Guidance  / 

Control  Verification 


Chase  Aircraft 

■  Both  Photo  and  Safety  Chase  Requirements 

■  Practice  Airborne  Pickup  With  Control  Room 

■  Chase  of  Surrogate  Aircraft  Good  Practice 

■  Consider  High  /  Low  Chase  for  “Agile”  UAV 

■  Thorough  Brief  With  Chase  Pilot  /  AVO  Required 

■  Chase-Control  Room  Comm  Is  Very  Important 
-Photo  Chase  KIO  Maneuver  Must  Be  Preplanned 
-“T”  Word  (Terminate)  Is  Not  to  Be  Used 


Contingency  Planning 


■  Develop  Useable  “What  If”  Procedures 

■  What  Ifs  Should  Lead  to  Limited  Preplanned  Options: 

-  Continue  on  Mission  Plan  (or  Orbit  /  Alternate  Route) 

-  RTB  (Following  Planned  Contingency  Routing)  Destruct 
(or  Let  It  Crash) 

■  Ground  Contingencies  Are  AVO  Memory  Items 

■  Defined  No  Fly  Zone  -  Remaining  Range  Is  OK  to  Fly 

■  Practice  in  Sim  with  All  Engineer  Stations  Simulated 

-  Develop  Teamwork  Attitude  and  Build 

-  Confidence“Practice  Like  You  Are  Going  to  Play  on 
Flight  Day” 


First  Flight/Lessons  Learned 

■  Plan  Short  Mission  With  Limited  Objectives 

■  Follow  Go  /  NoGo  Criteria,  FOLD,  Test  Cards 

■  Have  Plenty  of  Fuel  Onboard  for  Delays 

■  The  “Real  World”  Contingency  That  Happens  Will  Not 
Be  Anything  You  Planned 

■  Control  Room  Engineers  Are  Your  Best  WCA 

■  Remote  “Flying”  UAVs  Requires  Same  Concentration 
and  Test  Discipline  as  Piloting  Aircraft  (but  It  Will 
Never  Replace  the  “Real  Thing”!) 


Video  of  first  flight  of  Pegasus 
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Individual  Protection  in  a  CBRN 
Environment:  Case  Studies  of  the  Role 
of  T&E  in  Requirements  Generation 

Karen  McGrady 

karen.a.mcqradv@usnnc.mil 

Office:  703-617-2441 


Presented  at  the  24th  Annual  National  T&E  Conference 
Topic  area:  Requirements  Generation  Process 
and  How  the  T&E  Community  Can  Improve  It 
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Joint  Program  Manager  (JPM)  for 
Individual  Protection  (IP) 


JPM  IP  is  responsible  for  development,  procurement,  and 
sustainment  of  CBRN  environment  IP  for  the  Joint 


Services 

The  Joint  Service  General  Purpose  Mask  (JSGPM) 
system  has  been  developed  to  replace  existing 
masks  for  ground  and  shipboard  applications.  The 
JSGPM  will  provide  protection  against  chemical, 
biological  and  radiological  agents,  as  well  as  toxic 
industrial  materials.  It  is  being  designed  to  give  the 
wearer  the  widest  possible  field  of  view,  with  a 
lower  breathing  resistance  and  weight/bulk  less 
than  other  masks. 


The  Joint  Service  Aircrew 
Mask  (JSAM)  is  being 
developed  in  conjunction 
with  the  Joint  Protective 
Aircrew  Ensemble  (JPACE) 
to  replace  all  current 
aircrew  masks. 


Part  of  the  JSLIST  Block  I 
ensemble,  the  JB2GU  FR  will 
provide  hand  protection  from 
liquid,  vapor,  and  aerosol  CB 
hazards  for  users  with  a  flame 
resistant  requirement  (ground 
vehicle  crewmen,  SOF,  Small 
Boat  Teams). 
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JPM  IP  Case  Studies 


Resolution  of  conflict  between  requirements  and  tests 
designed  to  demonstrate  them: 

*  Quantifying  and  characterizing  wear  as  part  of 
chemical  permeation  testing  (JB2GU  FR) 

*  Ability  to  obtain  airworthiness  certification  in  order 
to  complete  OT  (JSAM) 

*  Quantifying  and  refining  the  TIC/TIM  threat  for  IP 
(JSGPM) 


T&E  Community  has  a  role  to  play  in  requirements 

generation 
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JSLIST  Block  2  Glove  Upgrade  Flame  Resistant 

(JB2GU  FR) 

*  Requirements  indicate  the  need  for  chemical 
protection  following  “wear”  hours 

*  Practice  has  been  to  use  representative  war 
fighters — piggy-back  training  opportunities  to 
maximize  “realistic”  wear  through  conduct  of 
mission  tasks 

*  Challenge  of  data  collection  and  control  of  test 
parameters/assets  during  a  training  rotation 


Wear  is  a  required  treatment  for  every  IP  system 
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JB2GU  FR  Wear  Testing 


•  Need  to  refine  wear 
requirements;  better 
quantification  and  data 
capture 

•  Not  all  wear  data 
created  equal 


JB2GU  FR  Glove  Wear  Profile  from  Ft.  Knox 


30 

25 


20 


15 


10 


5 

0 


Glove  Pair  TICN 


□  Weapons  Training 

□  Weapons  Maintenance 

■  Underway  Maintenance 

■  Tactical  Movement 

□  NVG  Patrolling 

□  Night  Navigation 

□  Map  Reading 

■  Equipment  Training 

□  Steering/Switches/Levers/Buttons 

□  Day  Navigation 

■  Briefing/Training 

□  Ammo  Handling 


JB2GU  FR  Glove  Wear  Profile  from  Wainwright 


140 
120 
100 

«/> 
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o 
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|  60 
40 
20 
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Glove  Pair  TICN 


□  Weapons  maint 

H  Tactical  movement 
H  Road  march 

■  Range  set-up 

□  Physcal  training 

□  Other 

□  Obstacle  Course 

■  Navigation  Movement 

■  Marksmanship 

□  Map  Reading 

□  Equip  Training 

□  Equip  Maint 

■  Dry  Fire  Drills/MAG 

□  Draw  Weapons 

□  Combat  Training 

■  Barrack  Maint 

□  Total  Admin 


Articulate  specific  type  of 
wear  acceptable  for  the  user. 
Consider  mechanical  “wear” 
as  part  of  developmental  test 
events 
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Lessons  Learned: 

JB2GU  FR  Wear  Testing 

•  Interpretation  of  the  wear  requirement 

-  Control  of  variables  for  subsequent  testing 

-  Increased  understanding  of  true  capability  of  the  system 
under  test 

-  Greater  assurance  to  user  community 

*  Realistic,  representative  wear 

•  More  powerful  results ,  increased  correlation  between 
stresses  applied  to  system  and  subsequent  system 
performance 
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Joint  Service  Aircrew  Mask  (JSAM) 

Key  performance  parameters  met — obstacles  to  OT 

execution: 

*  Air  worthiness  criteria  and  “safe-to-fly” 
certification — service  specific  requirements  for 
Navy,  Army,  Air  Force 

*  Multiple  configurations  on  multiple  platforms 
(rotary  and  fixed  wing) 

*  Structure  T&E  events  to  address  airworthiness  in 
increments  (configurations)  up  front  and  early 

Airworthiness  certification  required  to 
conduct  OT  for  aviation  systems 
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Lessons  Learned: 
JSAM 


•  Close  co-development  of  requirements  and  T&E 
structure  in  future  efforts 

-  Joint  Experimentation — field  representative  DT-type 
experiments  to  identify  integration  and  flight  issues  early  in 
program 

-  Collaboration,  cooperative  experimental  design,  analysis 


•  Airworthiness  criteria  addressed  upfront  and  early 

-  Structure  T&E  events  to  address  airworthiness  in  increments 
(configurations) 
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Joint  Service  General  Purpose  Mask 

(JSGPM) 

Emerging  threats  (TICs/TIMs)  impacting  systems  at 
all  points  in  the  acquisition  cycle 

•  Existing  guidance  has  been  the  ITF-25  list — not  all 
chemicals  have  the  exposure  potential  and  chemical 
properties  to  make  them  viable  threats 

*  JPEO  requirement  refinement — defining  TICs/TIMs 
of  interest  based  on  prioritization  strategy 

-  Cost,  schedule,  performance  control 


Lack  of  threat  definition  for  TIC/TIM  environment 
impacting  the  ability  to  verify  operational 

effectiveness 
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TIC/TIM  Task  Force 


Generate  specific  capability  area  lists 

Ultimate  Capability 


ITF-40 

J[  ITF-40  Table  ] 

- ►  ( 

C-2  +  Intel  A 

(1756  Chemicals) 

^  (~60  \y 

v  Chemicals)  J 

Acute  Toxicity/ 

Sense 
Shape 


Shield 


Sustain 


Likelihood 


*  Generate  vignettes  to  support  operational  hazard 
analysis,  CONOPS  development,  and  test 
planning. 
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Approach  to  Requirement  Refinement 


;  Industrial 

[  World 

j!  Uses 

fj  Output  1 
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Lessons  Learned: 

JSGPM 

•  Requirement  refinement  necessary  to  direct  future 
efforts 

-  Assure  prudent  investments  in  technology 

-  Better  characterize  current  system  performance 

•  Augment  redefinition  of  requirements  across  CB 
commodity  areas 

-  Approach  translates  to  development  of  materiel  solution  for 
detection,  collective  protection,  decontamination 

•  Direct  T&E  infrastructure  investments  for  future  test 
needs 
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Conclusion 


•  JPEO  CBD  directs  JPMs  to  actively  engage  in 
requirements  processes: 

-  To  insure  clear  understanding  of  the  operational  need 

-  To  prioritize  limited  resources 

-  To  understand  the  needs  for  technology  insertion  and  future 
capabilities 

-  To  insure  the  existence  of  adequate  test  methodologies  and 
facilities 


Our  successes  in  T&E  are  helping  to  refine , 
improve,  and  drive  requirements  generation 
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Point  of  Contact 


Primary: 

Dr.  Karen  McGrady 

Director,  Test  and  Evaluation,  Joint  Project  Manager  for  Individual  Protection 
(JPM  IP) 

Email:  karen.a.mcqradv@usmc.mil 

Phone:  703-617-2441 
Address:  Stafford,  VA  22556 

Alternate: 


Mrs.  Chris  Susman 
SURVICE  Engineering  Company 
Email:  chris@survice.com 

Phone:  410-273-7722 

Address:  4695  Millennium  Drive,  Belcamp,  MD  21017-1505 


14 


NORTHROP  GRUMMAN 


FINING  THE  FUTURE 


Lessons  Learned 
in  Fielding  a 
UAS  in  the 
Combat  Theater 

NDIA  24th  National  Test  and  Evaluation  Conference 
Palm  Springs,  California 

February  28th,  2008 

Sam  McKeehan 

System  Test  Engineer  -  Global  Hawk 
Northrop  Grumman  Corporation 


I  introduction 


/  have  been  deployed  five  times  in  support  of  the 
Global  War  on  Terror  since  September  11th  2001. 

Supported  Operation  Enduring  Freedom,  Operation 

Iraqi  Freedom,  and  Operation  Southern  Watch. 

I  supported  the  RQ4A  “ Global  Hawk”  High  Altitude 
Long  Endurance  aircraft  during  these  deployments. 

These  are  some  lessons  learned  that  have  since 
evolved  into  standard  operations  today. 


NORTHROP  GRUMMAN 
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Lessons  learned  Topics: 


A  1.  Pilot  Intervention 
fi  2.  Ground  Crew 
3.  Environment 
|  4.  Need  in  the  field 
I  5.  Risk  of  aircrew 


A  V3  Returning  Home 
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1.  Pilot  I  intervention 


K  ■  The  Fully  Autonomous  Air  Vehicle 

■  Pre  launch 

k/  ■  Mission  plan 

■  ■  Command  and  control 

I  ■  Command  Shelter 

■  ■  Hand  off 

■  ■  Aircrew  cycle 

I  ■  Pre  Brief/  Post  Brief 
I  ■  Identify  possible  issues 

■  ■  Clarify  mission  need 

■  Accurate  fault  or  discrepancy  description 
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1.1  The  Fully  Autonomous  Air  Vehicle 


Pre  Launch 

■  Preflight  A/C  and  Shelter 

■  Walk  around 

■  Outside  impacts 
Mission  Plan 

■  A/C  and  Shelter  Match 

■  Last  minute  updates 
Command  and  control 

■  LRE  (Launch  and  Recovery  Element) 

■  Control  Center 

■  MCE  (Mission  Control  Element) 
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1.2  Command  Shelter 


j 

i 


Hand  off 

■  LRE  launch 

■  MCE  Mission 

■  LRE  Recover 
Aircrew  cycle 

■  LRE 

■  MCE 
Crew  rest 
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1.3  Pre  Brief/  Post  Brief  (Pilots) 


■  Identify  possible  issues 

■  Intermittent  issues 

■  Scheduled  Maintenance 

■  Known  Comm/NAV  interference 

■  Clarify  mission  need 

■  SAR,  EO/IR 

■  Ground  CDL 

■  Accurate  fault  or  discrepancy  description 

■  Faults  during  mission 

■  Post  flight/mission  data  review 
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2.  Ground  Crew 


i 


Preflight  and  Post  flight 

■  Inspections 

■  Mission  plan 

■  Fault  logs 

Scheduled  Maintenance 

■  Down  time 

■  Mission  Cycle 
Pre  Brief/  Post  Brief 

■  Identify  possible  issues 

■  Clarify  mission  need 

■  Accurate  fault  or  discrepancy  description 
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2.1  Preflight  and  Post  flight 


i 


Inspections 

■  Electrical  (VTC) 

■  Mechanical 

■  Servicing 

Mission  plan 

■  Loading 

■  Check  sum 

Fault  logs 

■  Download  and  save 

■  Erase  for  new  mission 
Fault  Isolation 
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2.2  Scheduled  Maintenance 


■  Down  time 

■  Minor/Major 

■  Equipment 

■  Retest 

■  Mission  Cycle 

■  Schedule 

■  Impact  to  next 
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2.3  Pre  Brief/  Post  Brief  (Ground  Crew) 


Identify  possible  issues 

■  Recurring  faults 

■  Work  a  rounds 
Mission  cancel 


Clarify  mission  need 

■  Payload(s) 

■  Keying  Requirements 

Accurate  fault  or  discrepancy  description 

■  Trouble  shooting  time 
Fault  isolation 


Lead  time  on  replacements 
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3.  Environment 


■  Climate 

■  Maintenance  time 

■  Indoor/Outdoor 

■  Mission  Capable  Limits 

■  Equipment 

■  Availability 

■  Special  Equipment 

■  Part  replacement 

■  Hours  on  airframe 
Lead  Time. 
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3.1  Climate 


■  Maintenance  time 

■  Exposure  to  elements 

■  Coincide  with  mission 

■  24  hour  coverage 

■  Indoor/Outdoor 

■  Time 

■  Exposure  to  others 

■  Mission  Capable  Limits 

■  Take  off/  Landing 

■  Visibility  over  target 
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3.2  Equipment 


■  Availability 

■  On  site 

■  Loan  from  local  unit 

■  Sent  from  stateside 

■  Special  Equipment 

■  For  that  area 

■  Power  source  (s) 

■  COMSEC 
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3.3  Part  replacement 


Hours  on  airframe 

■  Increased  cycle 

■  Cost 

■  Expand  limits 
Lead  Time. 

■  Availability 

■  Customs 
Local  Vendor 


NORTHROP  GRUMMAN 
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4.  Need  in  the  Field 


Battlefield  Commander 

■  Near  real  time  Imagery 

■  BDA 

■  Re  Direct 
Troops  on  Ground 

■  More  Intel 

■  Direct  download 
Long  Endurance 

■  Time  in  Theater 

■  On  Task 

■  Flexibility 


Approved  for  Public  Release,  Distribution  Unlimited, 
WPAFB  08-0222  Dated  5  February  2008 


NORTHROP  GRUMMAN 


Copyright  2006  Northrop  Grumman  Corporation 


4.1  Battlefield  Commander 


TA  ■  Near  real  time  Imagery 

Accurate  ground  troop  placement 
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Camp  Fires  and  Cave  Entrances 


Taliban  Lookouts  on  Rid 


HEAT  SIGNATURE 


MIL  - 


Gunship  Round  Impacts 
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4.1  Battlefield  Commander 


■  Near  real  time  Imagery 

■  Accurate  ground  troop  placement 
Air  strike 
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Primary  Hits 


IR  Image:  BDA  from  AC-130  Gunship  Strike 
Tora  Bora  Cave  Complex,  AF  10  Dec  01  /  0200L 
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4.1  Battlefield  Commander 


■  Near  real  time  Imagery 

■  Accurate  ground  troop  placement 

■  Air  strike 
Potential  threats 
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4.1  Battlefield  Commander 


■  Near  real  time  Imagery 

■  Accurate  ground  troop  placement 

■  Air  strike 

■  Potential  threats 

■  BDA 

■  Verification 

■  New  targets 

■  Re  Direct 

■  Capable  to  look  outside  of  mission 

■  Determined  by  BDA 

■  T arget  of  opportunity 
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4.2  Troops  on  Ground 


■  More  Intel 

■  Local  Threats 

■  BDA 

Re  Direct 


■  Direct  download  (Demo) 

■  Hand  held's 

■  portable  ground  stations 
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4.3  Long  Endurance 

Time  in  Theater 

19  to  30  hours  (Average  24) 
Distance  from  target  area 

On  Task 

Coverage  of  active  mission 
■  Ongoing  BDA 


■  Flexibility 

■  To  complete  several  missions 

■  Support  other  Recon  aircraft 
Aircrew  limitations 


NORTHROP  GRUMMAN 
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5.  Risk  of  Air  Crew 


Pilot  Fatigue 

■  Long  Missions 

■  Crew  rest 

■  Crew  cycle 
Ground  Station 

■  Location 

■  Flexibility 

■  Divert  safety 
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Conclusion 


Performance  =95. 8%  single  A/C  turn  around  4  hour ; 
Efficiency  =91%  example:  40  consecutive  missions  100% 
Awards  =  Over  a  dozen  and  counting. 
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Questions 
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Cumulative  IOT&E  Results  Through  FY  2007 


Why  DOT&E  is  Focusing  on  Reliability 

DoD  O&S  Costs  -  Largest  Fraction  of  Life  Cycle  Costs 

Ground  Combat  Systems  Rotary  Wing  Aircraft 


3. 


Why  DOT&E  is  Focusing  on  Reliability 


"We  have  a  tendency  to  look  at  what  it  takes  to  get  a 
program  out  the  door.  We  don't  think  too  much  about 
what  the  life  cycle  [cost]  is.  It's  'Can  I  build  it?'  I  would 
like  us  all  to  be  mindful  of  what  it  costs  to  operate 
whatever  we  are  building  for  whatever  its  life  is  going  to 
be  because  I  have  to  pay  that  bill  every  single  year.“ 


(CNO,  ADM  Michael  G.  Mullen  in  an  interview  with  "Government 
Executive"  magazine  May  15,  2006) 


4. 


Empirical  Relationship  of  Studies 

Reliability  Investment  &  Reliability  Growth 


IDA  Data 


ATEC  Forward  Operational 
Assessment  Team 


AGENDA 

>  Mission 

>  Team  Composition 

>  Primary  Functions 

>  ATEC  Theater  Integration 

>  ATEC  FOA  Team  Facts 

>  Contributions 

>  Questions/Comments 


ATEC  FOA  Mission 


The  United  States  Army  Test  and  Evaluation  Command 
(ATEC)  Forward  Operational  Assessment  Team  deploys  to 
CENTCOM  to  conduct  liaison  with  MNC-I,  CJTF-XX,  CFLCC, 
and  CENTCOM  in  order  to  plan,  coordinate  and  integrate 
forward  assessment,  operational  testing,  evaluation  and 
experimentation  of  selected  systems. 


Personnel  total  24: 

Colonel-  1 
SGM  - 1 
Deputy  -  1 
CFLCC  LNO  -  1 
Logistics  - 1 
Information  Mngt  - 1 
Evaluator  -  2 
X-Pilots  (DTC)  -  2 
Data  Collectors  -  1 1 


Tech  Sppdcillsl  (DTC)  -  3 


-  Team  Deputy  /  Operations  Officer  (LTC) 

-  Information  Management  Specialist  (SFC) 

-  Evaluator  (AEC^  (Military  or  DAC^ 


MND-B 

MAJ 

NCO 


MND-N 

MAJ 

NCO 


COSCOM 

MAJ 

NCO 


OIF  Avn 


MNC-I 

CPT 

NCO 


Tech  cell 


DAC  (DTC-ATC) 
DAC  (DTC-EPG) 
DAC  (DTC-YPG) 


CJTF  XX 

MAJ 

NCO 


Alternates:  -  LTC 
-MAJ 

-  NCO  x2 

-  DAC 


2  x  Experimental  Pilots 
(DTC-ATTC) 


Primary  Functions 

•  Liaise  with  commands  in  theater 

•  Collect  data  and  report  on  specified  systems 

•  Conduit  of  information  between  theater  and 
CONUS 

•  Provide  ATEC  HQs  with  theater  priorities  for 
testing  in  CONUS 


A  TEC  Fwd  Integration 


JIEDDO-lraq 


CJTF  Troy 


MNC-I  C3  INTEGRATION 


AMC  FAST 


MNC-I  EWCC  (G) 
MNC-I  KMO 


HQDA  G8  LNO 


PEO  IEWS  LNO 


MNC-I  C-RAM  LNO 
JIEDDO-A  (TF  Paladin) 
JOINT  CREW  OFFICE 


Fielding  PMs 


ASA(ALT)  LNO 
HQDA  G8  LNO 


REF-Kuwait 


AMC-Kuwait 


CFLCC  C3 


CJTF  XX  FMD 


ATEC  FOA  Team  Facts 

•  ATEC  presence  in  theater  for  3+  years 

•  Collected  data  and  reported  data  on  200+ 
systems 

•  Significant  contributions  to  up-armoring  efforts 

•  Significant  contributions  in  EW  system 
performance  and  interoperability 


The  Most  Important  Contributions 


■  Getting  system  capabilities  and  limitations  to  leaders 


■  Linking  in-theatre  leadership  and  their  issues  back  to 
CONUS 

*  Testing  priorities 


■  Technical  expertise  on  the  ground  helping  to  solve 
challenges 

*  Ballistic  properties  of  armoring  solutions 

*  Electronic  countermeasure  system  support 

■  Total  ATEC  effort  to  support  the  theatre  on  armoring 
efforts 


COMMENTS/QUESTIONS 
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Improving  T&E  Participation  in  the 
Requirements  Generation  Process 


Chuck  Burdick  &  Keith  Montgomery 
Lockheed  Martin  Simulation,  Training  &  Support 

Paul  Bross 

Lockheed  Martin  Center  for  Innovation 

Presentation  #6448 

“The  Role  of  the  T&E  Community  in  the  Requirements  Process” 


24th  Annual 

NATIONAL 

TEST  &  EVALUATION 
CONFERENCE 

Requirements  Generation 

•  The  Government  regularly  conducts  analysis 
to  determine  requirements,  select  system 
capability  parameters,  perform  cost-benefit 
trade-offs,  etc. 

-  Analysis  is  a  critical  element  in  providing  capable, 
cost  effective  solutions  suitable  to  requirements 

•  To  support  this  analysis,  the  analytical 
community  maintains  several  models  and 
simulations  of  future  warfighting  threats  and 
capabilities 

-  Most  studies  are  tied  to  legacy  Service  models 

-  Threat  and  System  data  sets  are  updated  regularly 


Page  2 


24th  Annual 
NATIONAL 

TEST  &  EVALUATION 
CONFERENCE 

Requirements  Analysis  &  Simulation 

*  Typically,  for  an  OSD  sponsored  study  of  capabilities,  each 
Service  or  major  DoD  Agency  provides  one  or  more 
simulations  of  preference.  For  example: 

-  Thunder/STORM  for  the  Air  Force,  -  COSMOS/SEAS  for  Intel 

-  ITEM  for  the  Navy,  -  ELIST  for  Logistics 

-  JICM  for  the  Army  -  MIDAS  for  Strategic  Lift 

-  EADSIM  for  Missile  Defense  -  Pythagoras  for  SOF 

*  These  simulations  are  readily  available,  but  obtaining  the 
data  for  representing  friendly  forces,  threat  forces,  and  the 
situation  10  or  20  years  out  for  each  of  the  models  has 
been  a  challenge 

*  To  address  this,  the  OSD  Analytical  Community*  has 
developed  the  Analytical  Agenda,  a  collaborative  analytical 
framework,  to  examine  Defense  Strategy  and  Capabilities 
requ|remei|ts  along  wpa  common  source  of  credible 
data,  the  Joint  Data  Support  (JDS)  Office 


*  Office  of  the  Sec  of  Def  (Policy),  JCS  J8,  and  OSD  PA&E 
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Analytical  Agenda  Tenets 

*  Make  analyses  more  effective  and  relevant  for 
decision-making 

-  Focus  debate  on  assumptions  and  issues 

-  Help  synchronize  strategic  planning  activities  throughout  DoD 

*  Provide  a  common  starting  point  for  joint  analyses 

-  Scenarios  characterize  the  range  of  defense  challenges 

-  Future  joint  warfighting  CONOPS  are  developed/examined 

-  Analytic  assessments  (baselines,  data,  and  studies)  are  accessible  and 
well  documented 

*  Support  a  collaborative,  transparent,  and  continuously 
ongoing  analytical  process  with  the  Services, 
COCOMS,  Joint  Staff,  and  OSD  involved 

-  Train  and  sustain  a  critical  mass  of  analysts 

-  Managed  by  OSD  Policy,  Joint  Staff,  and  PA&E 


Create  a  Common  Simulation  Environment  for  Planning, 
Programming,  Acquisition,  Testing,  Experimentation,  and  Training 
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Analytical  Baseline  Process 

Current  Year: 


OPLANs 

Strategy  CONPLANs  Threat  CONOPS  Data 


Future  Year: 


National  OUSD  (P)  Joint  Staff  PA&E 

Leadership 
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Conducting  a  Study  with  Multiple  Models 


Having  common 
scenarios  and  data 
does  not  solve  all  the 
analytical  problems 


ISR 

Analysis 


COSMOS 


— u— 

—  mm 


Integrated  Air  Ops 


Thunder/STORM 


Most  studies  are 
conducted  using 
multiple,  incompatible 
models 


EADSIM 


BMD 
Analysis 


Wargames 
JICM 


Ground 

Warfare 


MIDAS 


Mobility 

Analysis 
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Analytic  Community  Models 
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M&S 
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Alternative  Analytical  Modeling  Methodology 


Integrated 
Air  Ops 

BMD 
Analysis 


Thunder/  Storm 


EADSIM 


Joint  Analysis  System 
Intearated  Camnaian 


Analysis 
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Integrated  Joint  Campaign  Modeling 

Analysis  Paradigm:  All  input  integrated  into  a  single,  joint  environment 


Basic  Scenario  Entities:  Forces  (US  Military,  Allies,  Neutrals,  Adversaries,  Terrorists,  US  Government 
&  Host  Nation,  Civil  Agencies,  NGOs/PVOs,  Populations) 

Intelligence,  Perception,  and  Knowledge  Base:  Multi-source  Collection,  prior  knowledge,  rule  sets 
Organization,  Directives  &  orders.  Communications)  -  C2:  Multi-actor  Info  sharing/  tiered 
dissemination,  Rules  of  Engagement,  Decision  Process,  Communications  Interoperability 
Response  Plans  (CONOPS):  Range  from  Facility  ->  Port  ->  Region  ->  National  ->  Coalition,  entities 
with  both  common  &  unique  capabilities 

Logistics  &  Support:  Support  for  Civil  Authorities;  constrained  transportation,  food,  fuel,  and  shelter 
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Agent-Based  Operations 

BSE  --  a  friendly  unit,  enemy  unit,  or  major  system  operating  in  the  battle  space.  Examples: 


Operational  Headquarters 
Support  Headquarters 
Airbases  and  seaports 
Infrastructure:  Power,  H20 
Civilians 


Land:  Units,  Neighborhoods 
Air:  Flights,  military  &  civil 
Maritime:  Ships,  small  craft 
Space:  Sensors  &  Comms 


Effects-based  and  much 
faster  than  Real  Time 


Command 
&  Control 


“Thinking,” 
Planning, 
Decision  Making 


Basic  Scenario  Entity 
(BSE) 


Detecting, 

recognizing, 

identifying 


Tracking  the 
BSE’s  assets 
and 

consumable 

resources 


Platform 


Communications 

Manager 


Owns  or 
Controls 


Location, 

speed, 

direction 


Communications- 
based  interface  to 
other  BSEs 


Event-based,  stochastic  model 
providing  detailed  cause  and 
effect  outcomes 
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JAS  C4ISR  is  Unique 


15 

1 

V 

■1 

k 

SENSORS 

•  Every  Sensor  input, 
including  Human 
Intelligence  is  recorded 

*  Ground  Truth  is 
simultaneously  recorded 
for  comparison 


Explicit 

COMMS 


•  Over  70  types  of 
operational 
messages  currently 
employed  (could  be 
increased) 

•  Uses  explicit 
networks  with 
capacities  and 
delays 


Updated 

erceptio 


•  Simulates 
information  fusion 

•  Common 
Operational  Picture 
maintained 

•  Sources  of  all 
information 
recorded 


Decisions* 


Actions 

•  Decisions  &  Actions  are 
traceable  to  their 
underlying  information 


k  •  Every  BSE  has  a  basic  Command  &  Control  capability  appropriate  to  the  BSEs 
purpose,  e.g.  explicit  headquarters  units,  artillery  unit,  supply  and  transportation 
unit,  etc.)  giving  a  set  of  allowable  decisions. 

•  A  BSE’s  C2  capabilities  are  further  enhanced  by  plug-ins  which  give  specific  new 
functionality  to  either  a  single  BSE  or  a  class  of  BSEs,  e.g.  a  Supply  Planner 

•  Knowledge  Bases  are  assigned  to  individual  BSEs  or  classes  of  BSEs  to  allow  them 
to  employ  user-selected  facts,  rules,  and  actions. 
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Joint  Data  Inputs 


Synthetic  Environment 


Solar/Lunar 
Atmospheric 
Sea/Sub  Surface 

•  Transmission  Loss 

•  Ambient  Noise 
Terrain 

•  Movement 

•  Line  of  Sight  p 


Raw 

Data 


Basic  Scenario  Entities 

•  Forces/Population  Centers 

•  Units/Civil  Groups 

•  Equipment 

•  Characteristics 

•  Interactions 

•  Performance 


CONOPS  ‘  Lme  ot  *'9™  Feeder 

•  Orders  &  Plans  Model 

•  Land/Air/Maritime 

•  ISR  Collection 

•  Any  Other  Object  Raw 

•  Formations  &  Positioning  Data 

•  Logistics 

•  Intelligence 

•  Doctrine 

Behaviors/Soft  Factors 


Raw 

Data 


Feeder 

Model 

■ 


Raw 

Data 


Deployment 

•  Origin 

•  Port  of  Embarkation 

•  Port  of  Debarkation 


Feeder 

Model 


Federation  “Hooks”  (as  rea’d! 


Training  Level 

Country  Of  Origin/Side  Changing 
Morale/Cohesion/Level  of  Commitment 
Degradation  of  Unit 


HLA  Certified/DIS  capable 

•  JSAF  Demonstrated 

•  Air  Defense  Demonstrated 
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JAS  Scenario  Bevels  of  Effort 


</> 

£  5 

c 

o 

B 

0)  ^ 

£  J 


'VT  MCO  - 1  (from  scratch) 
☆  MCO  -  2  (from  scratch) 


A 


Studies/Scenarios 

reusing 

.  MCO  Scenarios  M 


lA  MCO  -  2  (QVI Excursion) 

^fHLD  Vignette 
MCO- 1  mini-Excursions 


1 

Staff  Members/Subject  Matter  Experts 


Balanced 
Joint  Domains 

Air 

Maritime 

Land 

SOF 

Space 

with 

Communications 

ISR 

Logistics 


•  Large  Analytical  Baseline  Scenarios  builds  are  data  intensive  and  require  ~  9  persons  6  months 
to  build  whether  in  a  set  of  legacy  models  or  in  JAS 

•  However,  excursions  are  quick  using  existing  JAS  scenarios 

•  Reusing  an  existing  scenario  to  conduct  studies  and  evaluate  plans  requires  much  less  effort 

•  The  concept  is  to  maintain  the  major  scenarios  in  a  status  where  they  can  be  provided  in 
executable  format  to  any  interested  (and  approved)  user 
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Why  Test  in  a  Complex  Environment 

*  T&E  Community  can  quickly  perform  rigorous 
constructive  simulations  with  known  data  sets 

-  Enables  an  “Apples  to  Apples”  comparison 

-  JAS  covers  a  wide  range  of  realistic,  cross-functional 
scenarios 

*  Provides  the  ability  to  Test,  Remodel,  and  Retest  Prior 
to  first  LRIP  in  a  credible  analytic  scenario 

-  Cuts  production  and  pre-production  costs 

-  Allows  the  T&E  Community  to  identify  flaws  early 

»  Recommend  vital  changes  to  design,  and  rerun  the  simulation 

*  Offers  opportunity  to  apply  early  test  results  to  the 
original  National  Planning  Scenarios 

-  Assess  operational  implications  of  shortfalls  or  expanded 
capabilities 
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Time 


Field  Testing 
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Cost  of  change  rises 
exponentially  with  time 

JAS  provides  ability  to 
efficiently  model  the 
system  under  test  during 
Analysis  &  Design  phase 

-  Lowers  the  cost  of  change 

-  Increases  the  likelihood  of 
successful  acceptance 
test 

Enables  engineering 
community  to  create 
best  of  breed  solutions 


Page  15 


LOCKHEED  M  A  RtTn/, 


24th  Annual 
NATIONAL 

TEST  &  EVALUATION 
CONFERENCE 


Regional  Geography  Used  in  the  Trident 

Warrior  Wargames  •  Multiple  sides, 


Amberland 

{Amber) 


Jade 

(Onyx) 


Coral 


Amber 


Garnet 


Peridot  Isl. 
{disputed) 


Willard  Isl. 
{disputed) 


r'M  \i%i  yv\y 

Opal  Isl. 

(i Garnet)  %  \ 


Elbee  Isl 
(Garnet) 


•  Countries  and  non¬ 
state  BSEs  can 
change  sides, 

•  Considerable 
coordination  and 
communication  of 
military  and  civilian 
law  enforcement 


Illustrative  Findings 

•  Initial  level  of  Alert 
critical  to  outcome 

•  Most  plans  do  not 
address  an  adaptive 
opponent 
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Humanitarian  Support  Scenario 


UNHCR,  NGO  and 
Refugee  Camps 


Scene  of 
Earthquake 


Supply  Ships 


Aircraft 

Locations 


Advance  Base 


CuinrtTmtro  t3  ' 
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Controls 


VP  Control  Panel 


Resolution  (13  nuns) :  <  ||  j||~  |  t | 

Simulation  Time  (0  0  -  336  0)  H  ou«  :  2895 


JAS  Can  Federate  with 

Level  Models 

r  m,‘  JAS 


Red  forms  at  Line  of 
Departure  (LD)  for 
attack  on  Blue 


Campaign-level 
Aggregate  units 
Measures  of  Effectiveness 


♦ 

♦  * 

♦♦  ♦  ♦ 


Red  attempts  to  maintains 
line  formation  during  initial 
phase  of  attack 


Red  penetrates 
through  weak  areas 
in  Blue  lines 


JSAF 

*  Mission,  in  a  Campaign  context 

*  Views  of  Individual  Platforms 

*  Measures  of  Performance 
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JAS  as  a  Test  Context  Generator 


•  Covers  the  full  breadth  of  the  Theater  and  its  external  support  elements 

•  Uses  executable  scenarios  from  the  Analytical  Baseline  for  credibility 


Air  &  Missile 
Defense 
Detail 


Air2Air 

Detail 


Maritime 

Detail 


Ground 

Interaction 

Detail 


MITL 

Simulators 

Engineering 

Simulation 

Engagement 

Simulation 

Mission 

Level 

Model 

Federating  with  various  models  involves  some  effort,  but  once 
completed,  other  users  can  employ  the  same  interface 
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Summary 

•  The  T&E  Community  has  the  opportunity  to  participate  in  using  a 
common  analytical  simulation  framework  and  scenario  context  to 
potentially  unify  the  basis  of  requirements  generation, 
acquisition,  and  testing.  This  automatically  links  early  test 
simulation  effort  to  the  Defense  Planning  Scenarios  and  the  OSD 
Analytic  Baselines 

•  JAS  is  executable  GFE  software  and  comes  populated  with  data 
and  scenarios  (currently  10  available) 

-  While  scenario  development  from  scratch  is  hard,  excursions  from  existing 
scenarios  are  relatively  easy  to  run.  A  mix  of  both  Analytical  Baseline  (MCO-1 
and  MCO-2)  and  Title  10  scenarios  are  available  for  immediate  execution  and 
instant  credibility  with  minimum  effort.  Focus  can  be  on  the  proposed  tests 
and  evaluation,  not  on  building  scenarios. 

-  Cost  of  ownership  is  low.  Replications  generally  run  on  PCs  in  the  Windows, 
Unix,  or  Linux  operating  system.  The  larger,  more  complex  scenarios  require 
considerable  data  storage  and  servers  to  capture  all  the  output. 

-  HLA  federation  capability  allows  straight-forward  access  to  more  detailed 
models  and  human-in-the-loop  simulators  that  can  represent  proposed 
systems  to  be  tested  in  high  detail  in  a  full  theater  context. 
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What  are  the  Products 
of  an  Analytical  Baseline? 

•  Scenario  description 

•  Concept  of  Operations  (CONOPS) 

•  Joint,  Service,  and  Threat  Data 

•  Initial  assumptions,  methodology,  analytical  results,  insights,  and 
challenges 

•  Documentation,  usually  in  paper  or  electronic  format 

•  With  the  addition  of  JAS  to  the  family  of  analytical  models,  an 
analysis  can  also  produce  an  executable  and  maintainable 
simulation  of  the  specific  Scenario,  Plan,  or  Exercise  of  interest. 
This  can  form  the  basis  for  further  studies  or  excursions  using 
the  same  validated  scenario. 

-  Over  a  dozen  major  scenarios  are  either  completed  as  the  basis  for 
analytical  baselines  or  are  in  development. 

-  Three  of  these  MCO’s  have  been  translated  into  JAS. 
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DoD  M&S  Framework  and  JAS 


•Organized  by  Communities 

DoD  M&S  coordination  structured  to  support  the  Communities 
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Maritime  HLA  Example  from  a  Common  Scenario 


JAS  Aggregated 
Campaign  Level 

•  Combatant  locations  and  interactions 

•  C4ISR  interactions 

•  Weather/Sea  Conditions 


JSAF  Disaggregated 
Mission  Level 
Representation 


An  Enterprise  Environment  for 
Information  Assurance  / 
Computer  Network  Defense 
Testing  and  Evaluation 


Parker  Horner,  EWA  Gov’t  Systems  Inc. 
Steve  Moore,  Booz|Allen|Hamilton 
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Our  Thought 


•  IA-CND  testing  needs  an  enterprise  solution 

•  Network  enabled  operations  the  norm  and  growing  more 
robust  (by  the  day) 

-  Multiple  Mission  threads 

•  How  to  test  IA/CND  in  a  realistic  Mission  or  “Enterprise” 
environment??? 

-  Mission  Thread  equipment  and  manpower  fully  committed 


We’ll  explore  the  idea  of  using  the  10  Range  as 
a  launching  point  for  a  DOD  enterprise 
environment  for  IA/CND  testing 
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Underpinnings 


•  Definitional  and  Doctrinal  Discussion 

-  IA,  CNO,  CNE,  CNA,  CND 

-  Information  Assurance  — availability,  integrity,  authentication,  confidentiality, 
and  non-repudiation  ...  restoration  ...  protection,  detection,  and  reaction 
capabilities.  Note:  CND  provides  operational  direction  and  guidance  through 
global  network  operations  and  defense  for  employment  of  IA  in  response  to  a 
CND  alert  or  specific  threats. 

-  Computer  Network  Operations, —  Comprise  CNA,  CND,  and  related  CNE 
enabling  operations 

-  Computer  Network  Exploitation  (CNE)  —  Enabling  operations  and  intelligence 
collection...  adversary  automated  information  systems  or  networks 

-  Computer  Network  Attack —  disrupt,  deny,  degrade,  or  destroy  information 
...computers  and  networks  themselves 

-  Computer  Network  Defense  —  protect,  monitor,  analyze,  detect,  and  respond 
to  unauthorized  activity  within  DoD  information  systems  and  computer  networks 
...employs  IA  capabilities  to  respond  to  unauthorized  activity  ...employs 
intelligence,  counterintelligence,  law  enforcement,  and  other  military  capabilities 
to  defend  DoD  information  and  computer  networks 


(DoDD  0-3600.01,  Information  Operations,  August  14,  2006) 
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Policy  and  Direction 


Congress  and  the  2003  Appropriations  Act 
(the  IA  wake-up  call  for  Joint  Exercises) 

DOT&E  in  Nov  2006  -  IA  OT&E  policy 

-  End  to  End,  all  major  and  supporting  systems 

USD(AT&L)  and  DOT&E  in  Dec  2007  - 
operational  and  mission  context 

-  Mission  environment,  Projected  threat,  Life 
cycle 
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Joint  Operational  Environment 


What  is  in  it 

-  Platforms,  C2,  Data  Links,  ISR,  Data  management 

How  to  create  it 

-  Live  assets  very  limited 

-  Virtual,  constructive  not  geographically  collocated 

The  implied  challenges 

-  Realistic  operation,  interaction  of  elements 

-  Bring  the  environment  to  the  SUT 

Persistence? 

-  Ready  in  hours,  not  weeks 

-  Available  for  “test,  fix,  re-test”  cycling 

-  Consistent,  Knowledgeable 


JCAS  example 
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DoD  10  Range 

•  DoD  10  Roadmap  of  2003 

•  Nov  2005  DepSecDef  decision 

•  July  06  IOC  for  CNA  test  support 

•  Range  Architecture 

•  Current  Focus  of  10  Range 

•  The  Vision  for  the  10  Range 

-  Full  Spectrum  10 

•  Implied  Mandate  for  IA/CND 

-  Bringing  confidence  to  Joint  Force  Commanders 

•  Acquisition  Program  Requirements 

-  Need  growing 

-  Require  dependable,  repeatable,  expertise 

What  role  should  the  10  Range  play  in  DOD  IA/CND  testing? 
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Stakeholders 

USD(I),  ASD/NII 

-  IA/CND  must  be  delivered  to  warfighter,  not  fixed  in  the  field 

DOT&E 

-  Test  like  we  fight,  Realism,  Mission/Threat  representative 

JS  J5,  J3,  J6 

-  Want  COCOM  confidence  in  10,  IA,  CNO 

USD  (P&R) 

-  Consistency  between  testing  and  training  environment, 
CONOPS,  TTPs 

STRATCOM 

-  10  Warfighter  Support  Teams  must  be  part  of  Mission  Thread, 
CONOPS,  TTP  definition  process 

JFCOM 

-  “Service  Provider”  for  Joint  Test  and  Training  needs 

-  Consistency  a  must!! 
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Approach  and  Risks 


Cannot  step-function  jump  into  IA/CND 
Pilot  or  Proof-of-concept  required 
Leverage  successes  of  10  Range 

-  Handling  multiple  levels  of  classification 

-  Identify  what  changes  are  needed  for  IA/CND 

Rate  of  pace  of  change  and  stresses  that  places  on 
IA/CND 

-  75  +  potential  “IA/CND”  acquisition  Programs  in  the  FY08,  09,  10 
alone 

-  What  mission  threads? 

Is  there  enough  SME  to  support  the  IA/CND  workload? 
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Summary 


Framed  a  potential  DOD-wide  solution 
starting  point  for  IA/CND  testing 

Need  to  build  to  workload  and  confidence 
levels 

10  Range  -  “Provider  of  Choice” 

Look  to  continue  the  dialogue 
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Land  Systems 


Automatic  generation  of  Requirement 
Specifications  (Verification  Section)  -  in  DOORS 


Monika  Morgan 

Feb  08,  2008 
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Requirement  Specifications 


•  Department  of  Defense  requirement  specifications  are  primarily 
composed  of  six  sections  (per  MIL-STD-961D),  including  one  for 
Requirements  (Section  3)  and  another  for  Quality  Assurance 
Provisions  (aka  Verification)  (Section  4). 
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System  requirements  for  passenger  car 

1  INTRODUCTION 

2  APPLICABLE  DOCUMENTS 

3  REQUIREMENTS 

3^  Item  Definition _ 

>  3.2  System  Requirements 

>  3.3  System  constraints _ 

4  QUALITY  ASSURANCE  PROVISIONS. 

>  4.1  Qualification  requirements. 

>  4.2  Qualification  Methods 
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Verification  Section  (4.0) 


The  Verification  Section  is  composed  of: 
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Formal  module 


Edit  View  Insert  Link  Analysis  Table  Tools  User  GDL5  Help 
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El  - 1  INTRODUCTION 
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□  4.1  Qualification  requiremen 
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EJ-  4. 1.3  Design  qualificatioi 
+  4.1.4  Functional  testing. 
El-  4. 1.5  Acceptance  testim 
+  4. 1 .6  Test  conditions. 

+  4.1.7  Qualification  matri: 
-14.2  Qualification  Methods 
4.x  ...  (starting  at  4.2) 
5  PACKAGING 
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System  requirements  for  passenger  car  _ jjj  |  . 

4  QUALITY  ASSURANCE  PROVISIONS. 

4.1  Qualification  requirements. 

4.1 .1  Responsibility  for  inspection. 

4. 1.1.1  Inspection  equipment. 

4.1.2  Inspection  records. 

4.1.3  Design  qualification. 

4. 1.3.1  Qualification  plan. 

4.1.3.2  Test  plan. 

4. 1.3.3  EMR,  nuclear  hardening  test  plans. 

4. 1.3.4  Test  sequence.  ...  . .  ~ 

4.1.3.6  Final  reports.  Verification  Cross 

4.1.4  Functional  testing.  Reference  Index  (VCRI) 

4.1 .5  Acceptance  testing.  / 

4.1.6  Test  conditions.  / 

4.1.7  Qualification  matrix. 

4.2  Qualification  Methods-* - VeriflC3tion  , 

Methods 
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Verification  Section  -  VCRI  Matrix 
Template _ 

•  Within  this  table,  the  outline  for  Section  3  is  repeated  in  “Section  3  Par.” 

•  “Section  4  Par”  (Qualification  Methods)  paragraph  numbering  is  very 
similar,  usually  only  replacing  the  “3.”  with  “4.”. 
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Verification  Section  -  Verification 
Methods 


•  Each  object  is  covered  by  a  plan  for  testing  the  requirement 

•  It  may  be  written  with  a  “Shall”  as  a  test  requirement. 

•  Structure  of  this  section  follows  structure  for  Requirement  Section  (3.0) 


^formal  module 

1  File  Edit  View  Insert  Link  Analysis  Table  Tools  User  GDL5  Help 
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j  4.2  Verification  Method  Contents 
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* 

4.2  System  Requirements 


4.2.1  Functional  Requirements 

4.2.1 .1  Power  car 

4.2.1 .1.1  Motve  car 

4:2.1 .1 .1 .1  Move  forwards 

V  erify  that  the  c  ar  move  s  forwards  at  sp  e  e  ds  up  to  200  kilometers  p  er  hour  on  flat  ro  ads  as  sp  e  cifie  d. 

4:2.1 .1 .1 2  Move  backwards 

V  erify  that  the  c  ar  move  s  in  re  vers  e  at  sp  e  e  ds  up  to  20  kilometers  p  er  hour  on  flat  ro  ads  as  sp  e  cifie  d. 

42.1 .1 2  Accelerate  car 

V erify  that  the  car  can  be  accelerated  from  0  to  100  kilometers  per  hour  .... 

V erify  that  the  car  can  be  accelerated  from  0  to  150  kilometers  per  hour  .... 

V  erify  that  the  carcanbe  acc  elerate  d  from  0  to  200  kilometers  p  er  hour  .... 


Ld 

1  Username:  morganm 

Read-only  mode 

_ J 

Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-01 ,  dated  02/06/08  morganm@gdls.com 

GENERAL  DYNAMIC5 

Land  Systems 


The  Problem 


•  In  the  past,  these  documents  were  generated  manually, 
initially  using  typewriters  and  more  recently  word- 
processor  software. 

>-  Creating  and  maintaining  the  VCRI  matrix  using  available  table 
functions  can  be  very  cumbersome. 

►  Since  the  VCRI  table  and  the  Verification  Method  section  are  both 
organized  to  match  the  structure  of  Section  3,  their  contents  will  be 
affected  by  any  changes  in  structure  to  the  requirements. 

►  Some  Engineers  claim  that  the  requirements  and  section  structure  are 
stable  once  populating  Section  4.  However,  the  maintenance  is  still 
required  and  can  be  very  tedious  when  changes  do  occur. 

^  Even  when  using  a  requirements  management  tool  like  DOORS®,  a 
user  may  be  tempted  to  directly  populate  their  VCRI  and  Verification 
Method  sections  directly  in  Section  4. 

morganm@gdls.com 
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DOORS  For  Specifications 


•  Many  corporations  now  require  requirements  to  be  captured 
into  some  kind  of  database,  which  allows  the  capability  to 
show  traceability  to  customer  requirements. 

•  Our  company  has  chosen  Telelogic’s  DOORS®,  which  also 
allows  requirements  to  be  organized  to  resemble  the 
document  being  published. 

•  The  skeleton  for  the  document  is  organized  per  specific 
document  standards  and  generally  a  template  module  is 
created  to  contain  the  basic  structure  and  a  standardized  set 
of  attributes  and  views  to  the  data. 

•  The  template  is  then  used  for  creating  new  modules. 
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Key  Verification  Attributes 


•  The  “Verification  Method”  attribute  has  enumerated 
values  to  allow  the  user  to  indicate  how  a  requirement 
would  be  verified.  These  values  may  include: 

►  N/A  (used  for  non-requirement  objects,  including  headers) 

>-  Analysis 
>-  Inspection 
>-  Demonstration 
>-  Test 

•  If  the  attribute  is  set  up  as  a  multi-pick,  the  user  should 
make  the  best  single  pick  (there  may  be  certain 
situations  where  two  values  are  appropriate,  but  this  is 
the  exception). 
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Key  Verification  Attributes  -  Cont’d 


•  The  “Verification  Plan”  attribute  is  intended  to  contain  a 
sentence  or  two  on  how  the  requirement  is  to  be  tested 
and  may  be  written  as  a  test  requirement.  This  attribute 
is  not  intended  for  major  detailed  plans,  as  this  detailed 
information  would  be  handled  in  a  separate  Verification 
Plan  document. 

•  Tables  or  Figures  in  section  3  can  be  referenced 
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Key  Verification  Attributes  -  Cont’d 


•  Tables  or  Figures  that  are  unique  for  testing  can  be 
included  directly  in  Section  4.2. 

•  Besides  indicating  how  each  requirement  will  be  tested, 
establishing  values  for  both  of  these  attributes  also 
helps  to  isolate/correct  requirements  that  may  have  not 
been  written  well  (i.e.  not  testable). 
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Key  Verification  Views 


•  Three  specific  views  have  been  created  to  facilitate 
editing  and  reporting  the  values  of  these  attributes. 

►  “Verification  -  Editing” 

►  “VCRI  for  export  to  MS  Word” 

►  “Ver  Plan  for  export  to  MS  Word” 

•  These  3  views  provide  the  basic  layout  and  a  simple 
filter.  Specific  filters  can  be  created  when  exporting 
limited  requirements  for  a  specification. 


morganm@gdls.com 
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Key  Views  -  Editing  Attributes 


•  The  “Verification  -  Editing”  view  was  created  to  allow 
the  user  to  populate  verification  related  attributes  for 
each  of  the  requirements. 

•  Generally,  it  is  filtered  to  display  all  the  requirements  in 
section  3,  their  ancestor  parents  and  the  key 
verification  attributes  listed  above. 

•  Other  verification  type  attributes  that  will  be  used  for 
testing  down  the  line  may  also  be  included  in  this  view 
(status,  comments,  etc.) 
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Key  Views  -  Editing  Attributes, 
Cont’d 
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System  requirements  for  passenger  car  []  ' 

Verification  Method  ' 

Verification  Plan  — 

3R-255 

3  REQUIREMENTS 

— 

SR-25S 

3.1  Item  Definition 

SR-2  59 

3.2  System  Requirements 

SR-1 

3.2.1  Functional  Requirements 

SR-2 

3.2. 1.1  Power  car 

SR-3 

3.2. 1.1.1  Move  car 

SR-4 

3.2. 1.1. 1.1  Move  forwards 

SR -5 

The  car  shall  be  able  to  move  forwards  at  all  speeds  from  0  to  200 
kilometers  per  hour  on  standard  flat  roads  with  winds  of  0 
kilometers  per  hour,  with  ISO  BHP. 

Test 

V  erify  that  the  c  ar  move  s  forwards  at  sp  e  e  ds  up  to  200 
kilometers  per  hour  on  flat  roads  as  specified. 

SR -6 

3  2  1112  Move  backwards 

SR-7 

The  car  shall  be  able  to  move  backwards  to  a  maximum  speed  of 

20  Kilometers  per  hour  on  standard  flat  roads  with  winds  of  0 
kilometers  per  hour,  with  ISO  BHP. 

Test 

V  erify  that  the  c  ar  move  s  in  re  vers  e  at  sp  e  e  ds  up  to  20 
kilometers  per  hour  on  flat  roads  as  specified. 

SR-S 

3.2. 1.1. 2  Accelerate  car 

SR-Q 

Thfi  car  shall  ha  ahla  t.n  accalarat.a  frntn  n  t.n  inn  Kilntnat.firs 

T  ast. 

V  fififtr  t.h  at.  t.h  a  c  ar  c  an  h  ft  a  c  c  ftl  firat.fi  d  frntn  0  t.n  1  nn  T 

jlJ 

|  Username:  morganm  Read-only  mode 
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Key  Views  -  VCRI  Matrix 


•  The  “VCRI  for  export  to  MS  Word”  view  is  prepared  to  look 
like  the  intended  VCRI  matrix. 

•  Individual  columns  display  an  “X”  for  each  of  the  enumerated 
values  of  the  “Verification  Method”  attribute.  This  is  handled 
by  DXL  layout  code.  While  this  code  can  be  very  simple  and 
included  in  each  column,  a  customized  script  was  written 
that  provides  the  following: 

►  Ability  for  the  designer  to  pass  the  attribute  name  and  enumerated 
value  (set  up  for  multi-pick  enumerated  attributes). 

Verification  that  the  provided  attribute  name  exists  and  if  not,  displays 
an  error  directly  in  the  layout  column.  This  error  message  is  for  the 
designer’s  sake  and  is  not  intended  for  receipt  by  a  user. 

►  Automatic  classification  of  Header  Objects  as  "N/A“. 

morganm@gdls.com 
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Key  Views  -  VCRI  Matrix,  Cont’d 


•  The  “Section  4  Par”  column  displays  the  paragraph 
structure  for  Section  4.2,  which  corresponds  directly 
with  the  Section  3  paragraph  structure. 

•  This  view  is  exported  to  Microsoft  Word  (MS  Word)  as 
a  table  in  a  separate  document  and  then  is  added  to 
the  VCRI  template  (as  seen  earlier)  via  a  copy/paste. 
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Key  Views  -  VCRI  Matrix,  Cont’d 


1  Formal  module 

EM*. 

File  Edit  View  Insert  Link  Analysis  Table  lools 

User 

GDLS  Help 

Ib#^  jt  ^ 

B 

u 

■  — 
■  — 
■  — 

ti  ti  1 t 

I  1 

|  VCRI  For  Export  to  Word  |aII  levels  ^ 

(M3 

_ 

— 

==  = 

■  i 

E  1 

i  !s  H 

Y  £1 

Requirement  Paragra 

Title 

A 

i 

D 

Section  4  Paragraph 

;* 

3.2. 1.1 

Power  car 

X 

4.2. 1.1 

3.2. 1.1.1 

Move  car 

X 

4.2. 1.1.1 

— 

3.2. 1.1. 1.1 

Move  forwards 

X 

4.2. 1.1. 1.1 

3.2. 1.1.1. 1.0-1 

Move  forwards 

X 

4.2. 1.1.1. 1.0-1 

3.2. 1.1. 1.2 

Move  backwards 

X 

4.2. 1.1. 1.2 

3.2. 1.1. 1.2 .0-1 

Move  backwards 

X 

4.2. 1.1. 1.2 .0-1 

3.2. 1.1.2 

Accelerate  car 

X 

4.2. 1.1. 2 

3.2. 1.1.2 .0-1 

Accelerate  car 

X 

4.2. 1.1. 2.0-1 

3.2. 1.1.2 .0-2 

Accelerate  car 

X 

4.2. 1.1. 2.0-2 

3.2. 1.1.2 .0-3 

Accelerate  car 

X 

4.2. 1.1. 2.0-3 

3.2. 1.2 

Control  car 

X 

4.2. 1.2 

3.2. 1.2.1 

Switch  on  car 

X 

4.2. 1.2.1 

3.2. 1.2. 1.0-1 

Switch  on  car 

X 

4.2. 1.2. 1.0-1 

3.2. 1.2.2 

Control  speed 

X 

4.2. 1.2.2 

3.2. 1.2 .2.0-1 

Control  speed 

X 

4.2.1.2.2.0-1 

3.2. 1.2 .2.0-2 

Control  speed 

X 

4.2.1.2.2.0-2 

3.2. 1.2 .2.0-3 

Control  speed 

X 

4.2.1.2.2.0-3 

•w 

jd 

-ill 
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Key  Views  -  Verification  Plan 


•  The  “Ver  Plan  for  export  to  MS  Word”  view  is  prepared 
to  look  like  section  3,  showing  all  headers  and  text,  but 
the  text  in  this  case  displays  the  contents  of  the 
“Verification  Plan”  attribute. 

►  This  view  can  be  setup  to  display  all  objects  that  have  a  value 
in  this  attribute,  while  skipping  those  that  do  not,  and  can 
include  figures,  tables  and  their  captions. 

■  The  capability  to  display  figures/captions  recognizes  that  while 
figures  in  section  3  may  be  referenced,  some  cases  require 
special  values  in  the  tables  or  figures. 
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Key  Views  -  Verification  Plan, 
Cont’d 


While  this  view  displays  3  columns,  the  ID  and  requirement  columns 
are  provided  for  context  and  must  be  removed  before  exporting  the 
view  to  Microsoft  Word  in  Book  format.  The  results  of  the  export  would 
be  copy/pasted  into  section  4.2  of  the  Main  Document. 

Formal  module  [C~ ]fn~|fx~] 


File  Edit  View  Insert  Link  Analysis  Table  Tools  User  GDLS  Help 


Q 


Gff  X 


W* 


b  /  u  « 


4*  -if 


!\  I 


I  Ver  Plan  for  Export  to  Word  ^  |  |  All  levels  ~^\ 


\ 

□□c  c 


fS:  ^  lJ  V  ** 


ID 

System  requirements  for  passenger  car  []  ■ 

4.2  Verification  Method  Contents 

SR-2 

3.2. 1.1  Power  car 

4.2.1 .1  Power  car 

SR-3 

3.2. 1.1.1  Move  car 

4.2.1 .1.1  Move  car 

SR-4 

321111  Move  forwards 

4.2.1 .1 .1 .1  Move  forwards 

SR-5 

The  car  shall  be  able  to  move  forwards  at  all  speeds  from  0 
to  200  kilometers  per  hour  on  standard  flat  roads  with 
winds  of  0  kilometers  per  hour,  with  ISO  BHP. 

Verify  that  the  car  moves  forwards  at  speeds  up  to  200  kilometers  per 
hour  on  flat  roads  as  specified. 

SR -6 

3.2.1  1  1  2  Move  backwards 

4.2.1 .1 .1 2  Move  backwards 

SR-7 

The  car  shall  be  able  to  move  backwards  to  a  maximum 
speed  of  20  Kilometers  per  hour  on  standard  flat  roads  with 
winds  of  0  kilometers  per  hour,  with  ISO  BHP. 

V  erify  that  the  c  ar  move  s  in  re  vers  e  at  sp  e  e  ds  up  to  20  kilometers  p  er 
hour  on  flat  roads  as  specified. 

Jj 
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Key  Views  -  Verification  Plan, 
Cont’d _ 

•  The  Export  to  Word  function  can  be  extended  to  display  this  column  with 
the  same  styling  as  the  Object  Text/Object  Heading  column,  instead  of 


being  exported  as  attributes  of  Object  Text. 

^Export  To  Word  -  DOORS  [x] 

Formal  module 

General  Advanced  | 

p  Use 'Paragraph  Style' attribute  p  Use  Verification  Paragraph  Style' 

File  Edit  View  Insert  Link  Analysis  Table  Tools  User  GDLS  Help 

P  Allow  style  mapping 

P  Ignore  DOORS  table  borders,  put  all  borders  on  all  cells 

P  Ignore  DOORS  column  widths  in  table  layout  export,  AutoFormat  instead 

|P  Use  normal  template 

Template  name: 

1  y  m  li1 1  <  1 1  y  |  >  f  a* 

Ver  Plan  for  Export  to  Word  All  levels  3nC  nS?  d£.  SSSS 

4.2  Verification  Method  Contents 

4.2  System  Requirements 

42.1  Functional  Requirements 

| C:\Documents  and  Settings\morganm\My  Document  Browse...  j 

42.1.1  Power  car 

P  Use  horizontal  line  separators  in  book  layout  exports 

42.1.1.1  Move  car 

P  Save  Word  document  after  every  |  objects 

42.1 .1 .1 .1  Move  forwards 

P  Prepend  'Object  Identifier'  to  'Object  T ext'  (SRS) 

V  erify  that  the  c  ar  move  s  forwards  at  sp  e  e  ds  up  to  200  kilometers  p  er  hour  on  flat  ro  ads  as  sp  e  cifie  d. 

42.1 .1 .1 2  Move  backwards 

p  [Append  'Object  Identifier'  to  'Object  T ext'  (Requirements)  i 

V  erify  that  the  c  ar  move  s  in  re  vers  e  at  sp  e  e  ds  up  to  20  kilometers  p  er  hour  on  flat  ro  ads  as  sp  e  cifie  d. 

P  Append  Variants'  to  'Object  T ext'  (Requirements) 

42.1 .1 2  Accelerate  car 

p  Add 'N /A' to  empty  headings 

V erify  that  the  car  can  be  accelerated  from  0  to  100  kilometers  per  hour  .... 

V erify  that  the  car  can  be  accelerated  from  0  to  150  kilometers  per  hour  .... 

p  Center  images  on  page 

V  erify  that  the  carcanbe  acc  elerate  d  from  0  to  200  kilometers  p  er  hour  .... 

P  Center  tables  on  page 

P  Resize  larger  images  to  fit  on  page 

Username:  morganm  Read-only  mode 

Export...  ||  Close  Help 
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Ready  for  Export 


•  A  specification  is  handled  by  3  separate  exports: 

»-  Main  Document,  which  includes  the  contents  in  Sections  1-6 
>-  VCRI  Matrix 
>-  Verification  Plan 

•  The  export  can  be  handled: 

>-  Manually  -  manually  merged  and  polished 

>-  Semi-automated  -  A  single  script  could  be  developed  that 
performs  the  exports  and  merges.  Some  manual  polishing  will 
be  required  afterwards. 
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Export  Templates 


•  In  preparation  for  the  exports,  MS  Word  template  files 
(.dot  files)  have  been  created  to  direct  the  exports, 
provide  headers/footers,  a  cover  page,  a  Table  of 
Contents  and  style  formatting  for  the  whole  document. 
They  also  provides  caption  styles  for  Table  and  Figure 
captions. 

»-  Document  contents  and  Section  4.2  contents: 

■  Draft  Release  for  the  document  contents  (only  for  internal  reviews) 

■  Official  Release  for  the  document  contents 

^  VCRI  Matrix: 

■  Verification  Template 
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Exporting  a  Specification 


Main 

Document 

CIDS  Export  to  MS 
Word 

Official 

Release 

Book 

VCRI 

VCRI  for  export  to 

MS  Word 

Verification 

Table 

Verification 

Plan 

Ver  Plan  for  export 
to  MS  Word 

Official 

Release 

Book  (1) 

Ver  Plan  for  export 
to  MS  Word 

Official 

Release 

Table  (2) 

Notes:  (1)  Verification  Plan  exported  via  an  extended  Export  to  Word  function 

(2)  Verification  Plan  exported  via  the  “Out  of  the  Box”  Export  to  Word  function 
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Exporting  the  Main  Document 


•  A  view,  such  as  “CIDS  Export  to  Word”  would  include 
only  the  main  column  (Object  Text  and  Object  Heading) 

•  The  view  can  be  extended  with  the  desired  filter  based 
on  allocation  attributes,  if  necessary 

•  The  results  of  this  export  will  be  the  working  copy 
towards  a  completed  specification.  The  VCRI  and 
Verification  Plan  exports  will  be  merged  into  this 
document  within  Section  4.  The  resulting  MS  Word 
document  will  be  referred  to  as  the  “Main  Document”. 
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Exporting  the  Main  Document 
View 


Formal  module 


File  Edit  View  Insert  Link  Analysis  Table  Tools  User  GDL5  Help 


(T)(n)(x)  Export  To  Word  -  DOORS 


y  #  iS1 1  a  /  |r  iJb  i  a  |  ie  tc  y  es?  k 


CIDS  Export  to  Word  ^  1 1  All  levels  |  ilc  □!  |  ?£  L®j  V  zl  |  ^ 


System  requirements  for  passenger  car 


1  INTRODUCTION 

These  are  the  functional  system  requirements  for  the  development  of  a  new  passenger  motor  vehicle  as 
derived  from  the  user  requirements. 

The  car  will  have  a  world  wide  market. 

2  APPLICABLE  DOCUMENTS 

3  REQUIREMENTS 

3.1  Item  Definition 

3.2  System  Requirements 

3.2.1  Functional  Requirements 

3.2. 1.1  Power  car 

3 .2. 1.1.1  Move  car 

3.2. 1.1. 1.1  Move  forwards 

The  car  shall  be  able  to  move  forwards  at  all  speeds  from  0  to  200  kilometers  per  hour  on  standard  flat  roads  with 
winds  of  0  kilometers  per  hour,  with  ISO  EHP. 

3.2.1  1  1  2  Move  backwards 

The  car  shall  be  able  to  move  backwards  to  a  maximum  speed  of  20  Kilometers  per  hour  on  standard  flat  roads  with 
winds  of  0  kilometers  per  hour,  with  ISO  BHP. 

3.2. 1.1.2  Accelerate  car 

The  car  shall  be  able  to  accelerate  from  0  to  100  Kilometers  per  hour  in  10  seconds  on  standard  flat  roads 
with  winds  ofO  kilometers  per  hour. 

The  car  shall  be  able  to  accelerate  from  100  to  150  kilometers  per  hour  at  a  rate  of  5  kilometers  per  second 
on  standard  flat  roads  with  winds  of  0  kilometers  per  hour. 


jJ 
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General  Advanced  | 

p  Use 'Paragraph  Style1  attribute  |“  Use  Verification  Paragraph  Style1 
r~  Allow  style  mapping 

r~  Ignore  DOORS  table  borders,  put  all  borders  on  all  cells 

Ignore  DOORS  column  widths  in  table  layout  export,  AutoFormat  instead 
r~  Use  normal  template 
T  emplate  name: 

|gic  UG  2007\Templates\CiDS  Example  Template.dot  Browse...  | 

P  Use  horizontal  line  separators  in  book  layout  exports 
P"  Save  Word  document  after  every  I  objects 

Prepend  'Object  Identifier'  to  'Object  T ext'  (SRS) 

|  Append  'Object  Identifier'  to  'Object  T ext'  (Requirements) 

Append  Variants'  to  'Object  Text'  (Requirements)! 

P  Add 'N /A' to  empty  headings 
P  Center  images  on  page 
p  Center  tables  on  page 
P  Resize  larger  images  to  fit  on  page 


Export.. 


Close  Help 
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Exporting  the  VCRI  Matrix 


•  The  matrix  is  supplied  via  a  view  like  the  “VCRI  for 
export  to  MS  Word”,  which  can  be  tailored  for  specific 
specifications. 

•  The  view  is  filtered  to  only  show  requirements  and 
headers  in  Section  3,  normally  starting  at  3.2. 

•  The  view  is  exported  as  a  MS  Word  Table. 

•  Selecting  to  use  a  specific  export  template  like 
“Verification  Template.dot”  will  provide  an  empty  VCRI 
table  at  the  top  of  the  file  with  desired  formatting  for 
columns. 
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Exporting  the  VCRI  Matrix,  Cont’d 


Running  the  Export  to  Word  function 

4 1  Export  To  Word  - DOORS  (x) 


Layout:  . j  t  | 

Include  DOORS  Heading  Numbers 

|^  Issue  a  warning  when  an  unregistered  OLE  object  is  exported  as  a  picture 
|^  Include  Empty  Attributes 


Export...  !  Close  Help 


1 


Export  To  Word  -  DOORS 


General  Advanced  | 

Use  'Paragraph  Style1  attribute  |  Use  Verification  Paragraph  Style1 
I-  Allow  style  mapping 

\~  Ignore  DOGRS  table  borders,  put  all  borders  on  all  cells 

r-  Ignore  DOORS  column  widths  in  table  layout  export,  AutoFormat  instead 


I”  Save  Word  document  after  every  |  objects 

r-  Prepend  'Object  Identifier'  to  'Object  T ext'  (SRS) 
r-  Append  'Object  Identifier'  to  'Object  T ext'  (Requirements) 
r~  Append  Variants'  to  'Object  Text'  (Requirements) 
r-  Add  'N/A'  to  empty  headings 
Center  images  on  page 
|7  Center  tables  on  page 
W  Resize  larger  images  to  fit  on  page 

j  Export...  j  Close  Help 
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Exporting  the  Verification  Plan 


•  The  Verification  Plan  contents  is  supplied  via  a  view 
like  the  “Ver  Plan  for  export  to  MS  Word”,  which  can  be 
tailored  for  specific  specifications. 

•  The  view  is  filtered  to  show  requirements  and  headers 
in  Section  3,  normally  starting  at  3.2. 

•  The  Verification  Plan  export  needs  some  special 
handling.  Exporting  in  Book  format  normally  leads  to 
undesired  effects,  listing  the  column  as  an  attribute  for 
each  requirement  object. 
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Exporting  the  Verification  Plan, 
Cont’d 


•  The  Export  to  Word  script  can  be  extended  to  export 
this  column  as  if  it  were  the  main  column,  which  would 
then  assign  styles  to  Headers  and  captions  if 
tables/figures  are  included. 

•  The  final  results  of  this  export  are  in  a  new  MS  Word 
document  and  need  to  be  merged  in  to  the  Main 
Document  under  Section  4.2. 
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Verification  Plan  -  Better  Effects  as 
Book 


Exporting  via  “Extended”  Export  to  Word  function 


Export  To  Word  -  DOORS 


General  !  Advanced 


f  Use 'Paragraph  Style' attribute  F  Use  Verification  Paragraph  Style1 


P  Allow  style  mapping 


P  Ignore  DOORS  table  borders,  put  all  borders  on  all  ce,. 


P  Ignore  DOORS  column  widths  in  table  layout  export,  AutoForm^  ^tead 


P  Use  normal  template 
Template  name: 


|gic  UG  2007\Templates\CiDS  Example  Template. dot  Browse.. 
P  Use  horizontal  line  separators  in  book  layout  exports 


Save  Word  document  after  every 


P  Prepend  'Object  Identifier1  to  'Object  T ext1  (SRS) 


P  Append  'Object  Identifier'  to  'Object  T ext1  (Requirements) 


P  Append  Variants'  to  'Object  T ext'  (Requirements) 


F  Add  'N/A'  to  empty  headings 


F  Center  images  on  page 


W  Center  tables  on  page 


F  Resize  larger  images  to  fit  on  page 


Export...  Close 


^  Section  4.2  good  export.doc  -  Microsoft  Word 
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4.2.1  Functional  Requirements 

4.2. 1.1  Power  car 

4.2 .1.1.1  Move  car 

4. 2. 1 . 1 . 1 . 1  Move  forwards 

™  Verify  that  the  car  moves  forwards  at  speeds  up  to  200  kilometers  per  hour  on  flat  roads 
as  specified. 

4.2. 1. 1. 1.2  Move  backwards  T  | 

Verify  that  the  car  moves  in  reverse  at  speeds  up  to  20  kilometers  per  hour  on  flat  roads  ± 

m  as  specified. 

¥ 

Fa0:3.iJ  _ I  jJ 

Draw  -  fcfr  AutoShapes  -  \  \  □  O  (H)  ^  O  IS  H  -  £=*  O  [_  T 
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Summary 


•  While  there  may  be  some  overhead  on  the  export,  the 
end  result  leads  to  a  full  document,  without  the 
maintenance  drawbacks  of  needing  to  keep  the  section 
4  structure  current  with  Section  3.  Some  overhead  can 
be  diminished  by  automating  the  process. 

•  While  the  requirements  should  be  stable  once  Section 
4  is  being  populated,  there  is  still  the  opportunity  for 
change  and  re-arrangement. 

•  Therefore,  some  form  of  automation  for  Section  4  is  a 
worthwhile  endeavor  to  be  setup  by  the  DOORS  team 
for  any  company  that  needs  to  provide  consistent 
exports  of  specifications. 
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Appendix  -  Steps  to  Exporting  a 
Document 


•  The  following  steps  are  provided  for  exporting 
requirements  as  a  specification  in  more  detail. 

•  While  a  script  can  be  prepared  to  automate  these 
exports,  this  exercise  will  be  covering  the  manual 
approach 

•  We  have  a  customized  version  of  the  Export  to  Word 
function  provided  by  Telelogic.  Therefore  snapshots 
may  show  extra  features  that  won’t  be  seen  for  the 
original  function. 
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Export  from  DOORS  to  Word  - 
GDLS  Specific  Options _ 

•  GDLS  Extras  include  the  following  features: 

►  Append  the  Object  identifier  on  requirements 

►  Append  Variant  names  when  the  requirement  is  not 
Common? 

>-  Add  N/A  for  empty  sections 

>-  Browsing  for  a  customized  template  starts  at  the 
desired  folder,  instead  of  the  default  Microsoft  folder. 

►  Capability  to  print  Verification  Plan  information  in 
document  format  for  section  4.3.n 
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Export  a  CIDS  -  Export  Main 
Document 


Export  the  document 
body  from  the 
DOORS  module, 
including  all  headers 
and  requirement 
objects  within 
sections  1  through  6 
satisfying  the 
required  filter. 

Only  export  the 
column  containing 
the  heading  and 
object  text  attributes. 


Formal  module 


File  Edit  View  Insert  Link  Analysis  Table  Tools  User  GDLS  Help 


gff  |  a,  *  e  | ;?  |  x  ✓  |  r  ■«  |  b 


CID5  Export  to  Word 


U  abg  |  :=  |  !■*  [Sj1 

Tl  1 1  All  levels  |  d-c  *£  |  ==  ==  =  =  |  TS  ill  V  zi 


System  requirements  for  passenger  car 


1  INTRODUCTION 

These  are  the  functional  system  requirements  for  the  development  of  a  new  passenger  motor  vehicle  as 
derive  d  from  the  us  er  re  quirements . 

The  car  will  have  a  world  wide  market. 

2  APPLICABLE  DOCUMENTS 

3  REQUIREMENTS 

3.1  Item  Definition 

3.2  System  Requirements 

3.2.1  Functional  Requirements 

3.2. 1.1  Power  car 

3.2. 1.1.1  Move  car 

3.2. 1.1 .1.1  Move  forwards 

The  car  shall  be  able  to  move  forwards  at  all  speeds  from  0  to  200  kilometers  per  hour  on  standard  flat  roads  with 
winds  of  0  kilometers  per  hour,  with  ISO  BHP. 

3.2.1  1  1  2  Move  backwards 

The  car  shall  be  able  to  move  backwards  to  a  maximum  speed  of  20  Kilometers  per  hour  on  standard  flat  roads  with 
winds  of  0  kilometers  per  hour,  with  ISO  BHP. 

3.2. 1.1.2  Accelerate  car 

The  car  shall  be  able  to  accelerate  from  0  to  100  Kilometers  per  hour  in  10  seconds  on  standard  flat  roads 
with  winds  of  0  kilometers  per  hour. 

The  car  shall  be  able  to  accelerate  from  100  to  150  kilometers  per  hour  at  a  rate  of  5  kilometers  per  second 
on  standard  flat  roads  with  winds  of  0  kilometers  per  hour. 


Username:  morganm 


Read-only  mode 
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Export  a  CIDS 
Cont’d 

•  Initiate  the 

“Export  to  Word” 
function 
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-  Export  Main  Document, 


Formal  module  '/Work  A rea/Wo rganm/Verification  Exam p m  j^J [□] [~X 


Edit  View  Insert  Link  Analysis  Table  Tools  User  GDL5  Help 


T 


Save 


Ctrl+S 


Baseline 


Import 


I  X  V'  I  r  I<  I  B  I  U  ABC  |  :E  ti  tl 


n  ^ 


Module  Properties... 


Page  Setup... 
Print  Preview... 
Print... 


Ctrl+P 


icrosoft  Office 


HTML... 

Plain  Text... 

Rich  Text  Format.. 
Spreadsheet... 
FrameMaker... 
Interleaf... 


I  Word... 


Excel... 

Outlook. . . 
PowerPoint... 

;ne  development  of 

ser  requirements. 


Close 

3  REQUIREMENTS 

3.1  Item  Definition 

3.2  System  Requirements 


3CUMENTS 


3.2.1  Functional  Requirements 


3.2. 1.1  Power  car 

3.2. 1.1.1  Move  car 

3.2.1 .1.1 .1  Move  forwards 

The  car  shall  be  able  to  move  forwards  at  all  speeds  from  0  to  200 
kilometers  per  hour  on  standard  flat  roads  with  winds  of  0  kilometers  per 
hour,  with  180  BHP. 

3.2.1  1 .1  2  Move  backwards 

The  car  shall  be  able  to  move  backwards  to  a  maximum  speed  of  20 
Kilometers  per  hour  on  standard  flat  roads  with  winds  of  0  kilometers  per 
hour,  with  180  BHP. 

J  _  . 

|  Username:  testmlm  Read-only  mode 


,  GDLS  approved,  log  2008-01,  dated  02/06/08 
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Export  a  CIDS  -  Export  Main  Document, 
Cont’d 


pr  - - - 

ijExp 


Export  To  Word  -  DOORS 


General  i  Advanced 


Layout:  Book 


[7  Include  DOORS  Heading  Numbers 


|7  Issue  a  warning  when  an  unregistered  OLE  object  is  exported  as  a  picture 


17  Include  Empty  Attributes 


No  action  on  this  tab. 

Switch  to  the  Advanced  Tab 


Export  To  Word  -  DOORS 


General  I  Advanced 
[17  Use  'Paragraph  Style1  attribute 


P  Allow  style  mapping 


P  Ignore  DOORS  table  borders,  put  all  borders  on  all  cells 


P  Ignore  DOORS  column  widths  in  table  layout  export,  AutoFormat  instead 


x 
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Export  a  CIDS  -  Export  Main  Document, 
Cont’d 


For  large  documents,  its  helpful  to  select  “Save  document  after  every  100 
objects”  When  this  option  is  selected,  provide  a  file  name  once  prompted 
(see  next  page  for  file  name  specifics).  ixPOnjQ  word  -  doors  ~~  [ 


•Click  the  Export  button.  Minimize 
the  Microsoft  Word  window  once  it 
appears  on  screen  (This  process 
slightly  speeds  up  the  export  as  it 
doesn’t  have  to  keep  refreshing  the 
window) 

•If  you  get  a  prompt  about  a  style 
not  available,  click  OK. 


The  Word  style  name  'CaptionTable'  does  not  exist  in  this  template 
Standard  mappings  will  be  used  for  unknown  styles 


OK 


General  Advanced  | 

^  Use 'Paragraph  Style1  attribute 

P  Allow  style  mapping 

P  Ignore  DOORS  table  borders,  put  all  borders  on  all  cells 
P  Ignore  DOORS  column  widths  in  table  layout  export,  AutoFormat  instead 

P  Use  normal  template 
T  emplate  name: 

|gic  UG  2007\Templates\CiDS  Example  Template.dot  Browse...  | 

P  Use  horizontal  line  separators  in  book  layout  exports 


k  IS  ave  Word  document  after  every  I  |1 00  objects 


Select  to  Save 


Export... 


Close  Help 
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Export  a  CIDS  -  Export  Main  Document, 
Cont’d 


•  Save  the  file  twice  (The  file  name  should 
include  the  DOORS  module  being  exported 
and  the  date  that  it  was  exported  from 
DOORS): 

►  Save  the  newly  generated  Word  document  as  a  raw 
(untouched)  file  in  your  working  directory: 

►  Save  again,  but  as  a  working  file  to  contain  the  final  CIDS 
document  as  follows. 

•  The  working  document  will  now  be  referred  to 
as  the  “Main  Document”.  Minimize  this  window 
for  later  usage. 
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Export  a  CIDS  -  Export  VCRI  Matrix 


•  Export  the  VCRI  matrix  from  the  DOORS  module, 
including  all  headers  and  requirement  objects  within 
sections  3.0,  starting  at  3.2,  satisfying  the  required 
filter. 

•  Change  to  the  view:  “VCRI  Export  To  Word” 
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Export  a  CIDS  -  Export  VCRI  Matrix, 
Cont’d 


Formal  module 


File  Edit  View  Insert  Link  Analysis  Table  lools  User  GDL5  Help 


3EIE 


VCRI  For  Export  to  Word 


3.2. 1.1 


3.2. 1.1.1 

3.2. 1.1. 1.1 
1.1.1. 1.0-1 

3.2. 1.1. 1.2 
3.2. 1.1. 1.2 .0-1 

3.2. 1.1.2 


2. 1.1.2 .0-1 
2. 1.1. 2.0-2 
2. 1.1. 2.0-3 


3.2. 1.2. 1.0-1 
3.2. 1.2.2 
2.1 .2 .2 .0-1 
2. 1.2 .2 .0-2 


|  All  levels 


Verification  Method 


Power  car  X 


Move  car 
Move  forwards 
Move  forwards 
Move  backwards 
Move  backwards 
Accelerate  car 
Accelerate  car 
Accelerate  car 
Accelerate  car 
Control  car 
Switch  on  car 
Switch  on  car 
Control  speed 
Control  speed 
Control  speed 


Section  4  Paragraph 


4.2. 1.1 


4.2. 1.1.1 

4.2. 1.1. 1.1 

X  4.2. 1.1.1. 1.0-1 

4.2. 1.1. 1.2 

X  4.2. 1.1. 1.2 .0-1 

4.2. 1.1. 2 

X  4.2. 1.1. 2.0-1 
X  4.2. 1.1. 2.0-2 
X  4.2. 1.1. 2.0-3 

4.2. 1.2 
4.2. 1.2.1 
4.2. 1.2. 1.0-1 


Read-only  mod 
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Export  a  CIDS  - 
Cont’d 


•  Then  select  to 
export  to  Word 
using  able  option 
on  the  General  tab 
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port  the  VCRI  Matrix, 


Layout:  ^ 


1^  Include  DOORS  Heading  Numbers 

F  I  ssue  a  warning  when  an  unregistered  OLE  object  is  exported  as  a  picture 
17  Include  Empty  Attributes 


Export...  Close  |  Help 


)ved,  log  2008-01,  dated  02/06/08 


morganm@gdls.com 


40 


Export  a  CIDS  - 


•  Switch  to  Advanced 
Key 

•  De-select  to  “Use 
normal  template”,  and 
browse  to  select  the 
“Verification  Template” 
in  its  place. 

•  Click  the  Export  button. 
Minimize  the  MS  Word 
file  once  it  appears  on 
screen  (This  process 
slightly  speeds  up  the 
export  as  it  doesn’t 
have  to  keep  refreshing 
the  window) 

Approved  for  Public  Release,  Distribution  Unlimited,  GDLS 
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port  the  VCRI  Matrix 


Export  To  Word  -  DOORS 


General  I  Advanced  j 
|7  Use  'Paragraph  Style'  attribute 

F  Allow  style  mapping 

F  Ignore  DOORS  table  borders,  put  all  borders  on  all  cells 
F  Ignore  DOORS  column  widths  in  table  layout  export,  AutoFormat  instead 


F  Use  normal  template 
T  emplate  name: 

|lelogic  UG  2007\T emplatesWerification  Template.dot  Browse.. 


P  Use  horizontal  line  separators  in  book  layout  exports 
F  Save  Word  document  after  every;  |  objects 


rTxport'",""|  Close  j  Help 
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Export  a  CIDS  -  Export  VCRI  Matrix, 
Cont’d 


•  The  resulting  MS  Word  document  includes: 

»-  the  VCRI  table  template  as  an  MS  Word  table.  Optionally,  it 
may  be  divided  in  two  with  the  legend  as  separate  from  rest  of 
table. 

>-  the  exported  VCRI  contents,  in  a  separate  MS  Word  table. 

•  Save  the  file  as  a  raw  file  in  your  selected  working 
folder 

•  Scroll  down  to  the  exported  table 

•  Delete  the  top  row  that  contains  the  column  header 
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Export  a  CIDS  -  Export  VCRI  Matrix, 
Cont’d 


•  Select  all  columns  but  not  the  right  most  control  character  column 
and  select  to  Copy  (CTRL-C).  Do  this  by  moving  cursor  to  top  of 
table,  turning  to  black  arrow  and  dragging  across  rows. 

SysReqmts  VCRI  20Q7Sep4-raw.doc  -  Microsoft  Word  V][n](  X  | 


File  Edit  View  Insert  Format  lools  Table  Window  Help  Adobe  PDF  Acrobat  Comments 


i  J  lS  d  I  a  I 


Do 


50% 


jelA 


IT-  ¥ „l   |  ^  |  Final  Showing  Markup  t  Show  T  1^  t  Normal. NewMacros.Macrol 


S' 


. . . . . .  i  '  '  '  4 . 5  '  '  '  i  ■  ■  . . .  7  '  '  '  i  '  '  3  •  ■  ■  i  •  ■  ■  9  •  ■  ■  i  • 


=  Q 


Page  2  Sec  1  2/8 


English  (U.S 


EM  Z 

DF  Acrobat  Comment: 

5 

X 

1.  NewMacros.Macrol 

j. 

3 . .  .  1  .  .  .g  .  .  .  I  . . 

_ L 

A 
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Export  a  CIDS  -  Export  VCRI  Matrix, 
Cont’d 


•  Scroll  back  to  top  and  paste  the  VCRI  contents  from  the  clipboard 
into  the  leftmost  cell  of  the  first  empty  row  in  the  VCRI  table. 
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Export  a  CIDS  -  Export  VCRI  Matrix, 
Cont’d 
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Export  a  CIDS  -  Export  VCRI  Matrix, 
Cont’d 


Maximize  the  Main  Document  and  move  down  the  existing  stub 
VCRI  matrix.  Select  the  stub  VCRI  and  select  Paste  (CTRL-V). 

If  the  Legend  is  separate,  Copy/Paste  it  also  from  the  VCRI 
document  to  above  thjs  new  table,  keeping  as  a  separate  table. 

SysReqmts  2007Sep4-complete.doc  -  Microsoft  Word 


i  File  Edit  View  Insert  Format  lools  Table  Window  Help  Adobe  PDF  Acrobat  Comments 
j  i_3  £§?  I  (A  I  -  |  75%  ^  H  i  ^4  Times  New  Roman  -  1  |E  |  If  ,  m  |  :E  iW  tW 


IS  b i ^ L  =  =1  = Final 


Showing  Markup 


Normal ,  NewMacros .  Macro  1 


i  ^  •  •  2  ■  ■  ■  i  ■  ■  ■  3  ■  ■  i  ■  ■  ■  4  ■  ■  ■  i  ■  ■  ■  5  ■  ■ 

Table  II  Verification  Cross  Kefeien.ce  Index  (VCRI)  Matrix 


Legpnd 

Method 

N/A  Not  Applicable 
A  Analysis 
I  Inspection 
D  Demonstration 
T  Test 


Section  3  Paragraph 

Requirement  Title 

Method 

Section  4  I^ragraph 

NA 

A 

I 

D 

T 

3.2 

System  Requirements 

X 

4.2 

3.2.1 

Functional 

Requirements 

X 

4.2.1 

■J  '"I  1  1 

=  ts  Im  I  OP  < 


Ln  13  Col  1 


English  (U.5 


± 

o 

T 

> 
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Export  a  CIDS  -  Export  VCRI  Matrix, 
Cont’d 


•  Note,  the  new  table  now  follows  the  Table  Caption 
but  did  not  replace  the  stub  table. 

•  Scroll  to  the  end  of  the  new  table.  Delete  the  stub 
table  and  the  italicized  instructions  that  follow. 

•  If  the  heading  row  does  not  already  repeat,  select  the 
first  two  rows  and  select  Heading  Rows  Repeat  from 
the  Table  pull-down  menu.  May  need  to  click  to 
unselect  and  then  again  to  select  (as  a  toggle). 
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Exporting  the  VCRI  Matrix,  Cont’d 


Delete  the  instruction  text  in  the  main  document  and 
the  VCRI  template  block. 


Formal  module ' 


File  Edit  View  Insert  Link 

Analysis  Table  Tools 

User  GDLS  Help 

V  7 1  *•  » 

y  iv  i  t 

|  CIDS  Export  to  Word ▼ 

]  I  All  levels  ddc 

i  |  ?E  H  v  !i  |  Set 

System  requirements  for  passenger  c 


Demonstration.  An  uninstrumented  test  where  success  is  determined  on  the  basis  of  observations  alone. 

Test.  Instrumented  tests  verified  by  actual  measurement  that  the  equipment  meets  the  requirements  of  the  specification  when  subjected  to  the  actual  conditions 
(or  simulated  conditions)  specified. 

Table  IX  Verification  Cross  Reference  Index  (VCRI)  Matrix 


Verification  Legend 

Methpd_Categp_ry 

N/A  Not  Applicable  DV  Design  Verification 
A  Aniysis  FA  First  Article 

I  Inspection  A  Acceptance  tfflgL 

D  Demonstration  X  Component  Level  Verification  Mt.  |IB 

T  Test  g  fl  A  |  nil  | 


Section  3  Par. 

Requirement  Title 

Categpry 

FA 

A 

X 

Section  4  Par. 

The  contents  of  the  VCRI  table  just  above  will  bepMM  g  ^  K  A  to  Word"  view  in  DOORS.  DO  NOT  FILL  IN  THIS  TABLE,  it  is  for  export 
purposes  only.  This  table  should  not  be  dupiicatec^^J  a  julc  oe  duplicated  and  would  have  to  be  maintained  in  two  places.  The  ownerAisers  of  this 
module  should  use  the  "Verification"  view  to  fill  in ;  —  Verification  Method  attribute,  at  a  minimum.  This  attribute  is  used  to  create  the  actual  view  used for  a 
separate  export  to  produce  a  document.  The  filter  for  this  view  will  only  show  all  objects  within  sections  3. 0  through  3. 7  and  ignores  objects  marked  for 
deletion. 


5  PACKAGING 

J 

Username:  morganm 

Read-only  mode 

A 
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Export  a  CIDS  -  Export  Verification  Plan 


•  Change  to  the  “Ver  Plan 
for  Export  to  Word” 

view.  This  not  only  sets 
up  the  columns,  but 
filters  on  Section  3  only, 
where  the  Verification 
Plan  attribute  is  not 
empty. 

•  Delete  the  Object  ID  and 
Section  3  columns 
before  exporting, 

Leaving  only  the  “4.3 
Verification  Method 
Contents”  column  only. 


Formal  module  '/Work  Area/Mo rganm/Verification  Examples  -  UGC2007/Sm  (■"]["□]  fx] 


4.2.1 .1  tower  car 

4.2.1  .l.llVIove  car 


3.2.1  Functional  4.2.1  functional  Requirements 

Requirements  \ 

3.2. 1.1  Power  car  4.2.1 .1  tower  car 

3.2. 1.1.1  Move  car  4.2.1  .l.lWove  car 

3.2.1  1 .1  1  Move  forp-J-  J  *  111  \" —  f - J- 

The  car  shall  be  able  to  n  ^  r  j  j  j  j  : ! .  j  j  j  ^  LI  u .  ^  j  n  urn:  . 

forwards  at  all  speeds  frc  File  Edit  View  Insert  Linll  Analysis  Table  Tools 
kilometers  per  hour  on  st  user  GDL5  Help  \ 

roads  with  winds  of  0  kilt  7— — — — ~~r  — \~ — 

hour,  with  180  BHP.  ‘S'  ^  I  I  V 


Username:  test 


Ver  Plan  for  Export  to  Word  lAll  levels  ^  |  ic  □! 


4.2.1  Functional  Requirements 

4.2.1 .1  Power  car 

4.2.1 .1.1  Move  car 

4.2.1 .1.1.1  Move  forwards 

Verify  that  the  car  moves  forwards  at  speeds  up 
to  200  kilometers  per  hour  on  flat  roads  as 
specified. 


m 


Username:  testmlm 


Read-only  mode 
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Export  a  CIDS  -  Export  Verification  Plan, 
Cont’d 

Select  Export  to  Word,  via  an 
extended  version  of  the  function. 

►  General  tab:  no  actions 

►  Switch  to  the  Advanced  tab; 

►  Select  “Use  ‘Verification 
Paragraph  Style’  feature. 

►  Uncheck  “Use  normal  template” 

>-  Browse  and  select  the  same 
template  you  used  for  the  Main 
Document  to  apply  same  styles 
to  headers  and  text. 

►  Export.  A  new  MS  Word 
document  will  be  opened. 


morganm@gdls.com 


r - 

I  Export 


To  Word  -  DOORS 


[7  Use  Verification  Paragraph  Style1 


General  Advanced  | 

[7  Use 'Paragraph  Style1  attribute 
f~  Allow  style  mapping 

P  Ignore  DOORS  table  borders,  put  all  borders  on  all  cells 
r  Ignore  DOORS  column  widths  in  table  layout  export,  AutoFormat  instead 
Use  normal  template 
Template  name: 


|gic  UG  2007\Templates\CiDS  Example  Template. dot 


Browse.. 


|H  Use  horizontal  line  separators  in  book  layout  exports 
r  Save  Word  document  after  every  P”  objects 

P  Prepend  'Object  Identifier'  to  'Object  T ext'  (SRS) 

Append  'Object  Identifier'  to  'Object  T ext'  (Requirements)  I 
P  Append  Variants'  to  'Object  T ext'  (Requirements) 

[7  Add  'N/A'  to  empty  headings 
[7  Center  images  on  page 
[7  Center  tables  on  page 
[7  Resize  larger  images  to  fit  on  page 


Export.. 


Close  Help 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-01,  dated  02/06/08 

GENERAL  DYNAMICS 

Land  Systems 


50 


Export  a  CIDS  -  Export  Verification 
Plan,  Cont’d 


•  Save  the  newly  generated  Word  document  as  a  raw 
(untouched)  file  in  your  selected  working  folder 

Scroll  down  to  the  text  containing  the  Verification 
Plan  sections,  select  all  of  sections  from  4.3.1  to  the 
last  section  and  copy  (  RL-C  ). 

•  Maximize  the  Main  Document  window,  paste  (CTRL- 
V)  starting  after  the  header,  “4.2  Qualification 
Method”. 

Remove  the  italicized  instructions,  if  not  removed 
during  previous  instructions  for  the  VCRI  table 
export. 
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Export  a  CIDS  -  Export  Verification 
Plan,  Cont’d 


IS  SysReqmts  2G07Sep4-raw.doc  -  Microsoft  Word 


BE® 


File  Edit  View  Insert  Format  Tools  Table  Window  Help  Adobe  PDF  Acrobat  Comn 
J  G2  id  uJ  I  dt  lA  I  'SP  I  -J  B  I  3  -  I  ft,  I  100%  -  1  :  ii  Times  New  Roman 


J  $2  Q  lA  I  ^  lA  I  ^  I  - J  B  I  1^1  100%  -  H  j  44  Times  New  Roma 

Final  Showing  Markup  -  Show  T  rf  ■  J'  T  ^ 


SysReqmts  2007Sep24-corrected.doc  -  Microsoft...  -  □  X 


4;2  OuaJUficatWh  Methods 


The  contents  of  the  Verification  Pla, 
view  in  DOOP^l  The  headers  from  , 
duplicated  and  would  have  to  be  mi 
" Verification "  wew  ire  Ver 

actual  view  used  for  a  separate  exp 
with  an  assigned  Verification  Plan  - 
columns  from  this  view  before  expo , 
("4. 3  Verification  Method  Contents 


e  provided 


o  [5]  $)  < 

ge  13  5ec  3 

15/18 

At 

Ln 

Col 

EXT  OVR  English  (U 

Final  Showing  Markup 


.  .  2  ■  ■  ■  i  ■  ■  ■  3  ■ 


Replaces  Italic  Text 


14.2.1  Functional  Requirement? 

4.2. 1.1  Rraer  car] 

4.2.1.1.1  Move  car 

4.2. 1.1. LI  IVbve  forwards 

Verify  that  the  car  moves  forwards  at  speeds  up  to  200  kilometers  per  hour  on 

4t2.1.1.L2  Move  backwards 

Verify  that  the  car  moves  in  reverse  at  speeds  up  to  20  kilometers  per  hour  on 

4t2.1.1.2  Accelerate  car 

Verify  that  the  car  can  be  accelerated,  from  0  to  100  kilometers  per  h 


Verify  that  the  car  can  be  accelerated,  from  0  to  150  kilometers  per 
Verify  that  the  car  can  be  accelerated,  from  0  to  200  kilometers  per 


_ A  1  h>  rn__j_ 

=  B  |E|  "3  ip  < 
Page  17  Sec  3 


22/28  At  1.5"  Ln  3  Col  18 


IB 
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Alternative  Exports  for  Verification  Plan 


•  Besides  the  capability  of  extending  the  Export 
to  Word  function,  the  following  methods 
described  in  the  following  slides: 

►  Export  as  a  Table 

►  Export  as  a  Book  (Undesired  results,  not 
recommended) 
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Verification  Plan  -Effects  as  Table 


•  The  “4.2  Verification  Method  Contents”  column  could 
be  exported  as  a  Table  to  MS  Word  and  then  converted 
to  text. 

►  No  styles  are  provided  for  Headers  and  would  have  to  be 
applied  by  hand. 

►  Can  be  done  easily  via  the  Style  Formatting,  but  still  takes 
time. 
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Verification  Plan 


•  Exporting  via  “Out 
of  the  Box”  Export 
to  Word  function  as 
a  Table. 
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Effects  as  Table 


SysReqmts  Ver  Plan  2007Sep4-table  format-raw.doc  -  Microsoft  W...  |C~|fn~|fx~| 


[  File 

Edit  View  Insert  Format  Tools  Table  Window  Help  Adobe  PDF 

Acrobat  Comments 

X 

IDiBB  iay  IT  100%  - 

» 

4l  Normal  -  12  •  B  »]•  «  §E  :=  it  it  ^  •  A  - 
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Final  Showing  Markup  ^  Show  T  t  Normal.  NewMacros.  Macro  1 
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4.2  Verification  Method  Contents 

4.2  System  Requirements 

- 

4.2.1  Functional  Requirements 

— 

4.2.1  1  Power  car 

4.2. 1.1.1  Move  car 

4.2. 1.1. 1.1  Move  forwards 

Verify  that  the  car  moves  forwards  at  speeds  up  to  200  kilometers 

- 

per  hour  on  flat  roads  as  specified. 

4.2. 1.1. 1.2  Move  backwards 

- 

Verify  that  the  car  moves  in  reverse  at  speeds  up  to  20  kilometers 
per  hour  on  flat  roads  as  specified. 

4.2. 1.1.2  Accelerate  car 

- 

Verify  that  the  car  can  be  accelerated  from  0  to  100  kilometers  per 
hourn^ 

LI 

Verify  that  the  car  can  be  accelerated  from  0  to  150  kilometers  per 
hour^^ 

Verify  that  the  car  can  be  accelerated  from  0  to  200  kilometers  per 

hour^. 

4.2. 1.2  Control  car 

4.2. 1.2.1  Switch  on  car 

i 

Verify  the  car  recognizes  authorized  people. 

Tl 

4.2. 1.2.2  Control  speed 

± 

Verify  by  inspection  that  a  food  mechanism  is  present  to  be  able  to 

Q 

control  the  speed  of  the  car. 

¥ 

1  Draw 1 

r  fe  AutoShapes  -  \  ^□0|i|4vSlH  &  T  'JL  T  A,  T  = 

> 

|  Page 

1  Sec  1  1/3  At  1"  Ln  1  Col  1  1 

EnJ 
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Verification  Plan  -Effects  as  Table 


•  Headers  need  to  be 
formatted  as  MS  Word 
Heading  styles 


% SysReqmts  Ver  Plan  2007Sep4-table  format  to  text.doc  -  Microsoft  Word 


File  Edit  View  Insert  Format  Tools  Table  Window  Help  Adobe  PDF  Acrobat  Comments 


Jl®<?  "  -  &,  ffl  □  51 11  IT  100%  -  ©  ” 

|  Heading  3  -  Times  New  Roman  -  12  -  bJ  1  W1 |_  *  ~  ^  '  A  '  ” 

Final  Showing  Markup  -  Show  ’  *^  ^  ^  *  ^  *  ^  ’  1|^  ffl  Normal .  NewMacr os ,  Macro  1  , 


16  S’. 

. 

■  _ 


1.2.1  Functional  Requirements 


42.1.1  Power  car 

4. 2. 1.1.1  Move  car 

42.1.1.1.1  Move  forwards 

Verify  that  the  car  moves  forwards  at  speeds  up  to  2 ( 
as  specified. 

42.1.1.12  Move  backwards 

Verify  that  the  car  moves  in  reverse  at  speeds  up  to  2 
as  specified. 


4.2. 1.1.2  Accelerate  car 

Verify  that  the  car  can  be  accelerated  from 
Verify  that  the  car  can  be  accelerated  from 
Verify  that  the  car  can  be  accelerated  from 

42*12  Control  car 


0  to  100 1 
Oto  150 1 
0  to  200  i 


^  Styles  and  Formattim  ▼  X 
Formatting  of  selected  tewt 


Select  All  New  Style 

Pick  formatting  to  apply 

E-mail  Signature  H  - 


Envelope  Return 


Heading  1 
Heading  2 

Heading  3 

Heading  4 


4.2. 1.2.1  Switch  on  car 

Verify  the  car  recognizes  authorized  people. 


Heading  5 


4.2. 1.2.2  Control  speed  „  ,  — — 

*  Show:  Available  formatting  ▼ 

1  _ |  ±1 

Draw  -  AutoShapes  -  \ 


Page  1  Sec  1 


At  1"  Ln  1  Col  1  English  (U.S 
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Verification  Plan  -  Undesired  Effects  as 
Book 
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Export  a  CIDS  -  Remaining  Tasks 


Prepare  Front  Page  supplying  info  for  the  following 
items: 

►  Subsystem/component  name  for  module 

•  Update  the  page  headers  for  unique  MS  Word  sections 
to  include  name  of  subsystem/component  represented 
(e.g.  engine) 
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Export  a  CIDS  -  Remaining  Tasks, 
Cont’d 


•  For  Tables  and  Figures:  Captions  should  be  on  the  same 
page  as  the  table  or  figure:  table  caption  before  the  table; 
figure  captions  after  the  table.  If  possible,  keep  the 
referencing  object  with  the  table/figure  and  caption,  allowing 
all  3  objects  to  be  on  one  page.  This  can  be  handled  by 
manual  page  breaks  or  setting  the  paragraph  style  for  the 
referencing  requirement  to  be  “Keep  with  Next”. 

•  Update  the  Table  of  Contents,  Table  of  Tables  and  Table  of 
Figures  as  one  of  last  steps. 

•  Switch  to  the  Advanced  Tab,  no  changes  on  General  tab. 
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Monika  L.  Morgan 
General  Dynamics  Land  Systems 

morqanm@qdls.com 

586-825-5685 
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Integrating  Test  and  Evaluation 
(T&E)  Into  DoD  Acquisition 

Contracts 

Ms.  Darlene  Mosser-Kerner 

T&E  Policy 


Developmental  Test  &  Evaluation 
OUSD(AT&L)/Systems  &  Software  Engineering 
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DEVELOPMENTAL  TEST  &  EVALUATION 


Outline 


*  The  Challenge 

*  Test  and  Evaluation  (T&E)  Contract  Issues  Pre- 
Solicitation 

*  T&E  Contract  Solicitation  Issues 

*  T&E  Contract  Execution  Issues 

*  Integrating  T&E  in  Acquisition  Contracts 

*  Summary 


2 


T&E  -  From  Concept  to  Combat 


DEVELOPMENTAL  TEST  &  EVALUATION 


The  Challenge 


*  T&E  expertise  is  applied  late  -  during  source 
selection  or  after  contract  award 

*  Leads  to  contract  related  issues 

-  Lack  of  appropriate  contract  requirements 

-  Inappropriate  contract  incentives 

-  Lack  of  adequate  /  appropriate  test  resources 

-  Lack  of  adequate  test  time 

*  How  can  the  T&E  community  help  get  better 
contracts  developed,  awarded,  and  executed? 
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T&E  -  From  Concept  to  Combat 


Pre-Solicitation  T&E  Issues 

DEVELOPMENTAL  TEST  &  EVALUA TION 

•  Requirements  and  Strategies 

-  Development  of  evaluatable  requirements 

-  Development  of  an  appropriate  T&E  strategy 

•  Establishment  of  appropriate  contract  incentives 

-  De-emphasize  single  point  test  events  -  “first 
flight” 

-  Demonstrate  maturation  of  system  and  T&E 
program 

•  Scoping  the  test  program 

-  Timing  and  magnitude  of  the  test  effort 

-  Number  of  test  articles 

-  Threats,  targets 

T&E  -  From  Concept  to  Combat 
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Solicitation  T&E  Issues 

DEVELOPMENTAL  TEST  &  EVALUA TION 


*  Development  of  the  Statement  of  Objectives  (S00) 
and  Statement  of  Work  (SOW) 

*  Development  of  valid  source  selection  evaluation 
factors  and  scoring  criteria 

*  Participation  on  source  selection  teams 
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T&E  -  From  Concept  to  Combat 


Contract  Execution  T&E  Issues 

DEVELOPMENTAL  TEST  &  EVALUA TION 


*  Integration  of  Contractor,  Government,  and 
Operational  teams  -  Integrated  Test  Team 

*  Working  with  Defense  Contract  Management  Agency 
(DCMA) 

*  Test  Operations 

-  Sharing  data 

-  Deficiency  identification  and  resolution 

-  Safety  concerns 

-  Environmental  issues 

-  Resource  management 
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T&E  -  From  Concept  to  Combat 


System  Test  Plan  (STP) 


DEVELOPMENTAL  TEST  &  EVALUATION 


Integrating  T&E  In  Contracts  -  Ideas 

DEVELOPMENTAL  TEST  &  EVALUA TION 


*  Develop  T&E  strategy  (draft  TEMP)  prior  to  Request 
For  Proposal  (RFP)  development 

*  Ensure  requirements  are  evaluatable 

*  Use  Contractor  T&E  approach  in  source  selection 
criteria 

*  Consider  Test  execution  issues  in  SOO/SOW 

-  Integrated  CT,  DT,  OT 

-  Information  sharing 
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T&E  -  From  Concept  to  Combat 


Next  Steps 

DEVELOPMENTAL  TEST  &  EVALUA TION 


*  Engage  with  program  management  and  contracting 
communities 

*  Publish  T&E  contracting  guidebook  -  Summer  08 

*  Request  your  inputs  to  make  the  guidance  useful  to 
all  concerned 
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T&E  -  From  Concept  to  Combat 


Contact  Info 

DEVELOPMENTAL  TEST  &  EVALUA TION 


Darlene  Mosser-  Kemer 


Contact  us  to  provide  feedback  and  share  your  experience 
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T&E  -  From  Concept  to  Combat 


Air  Force  Research  Laboratory 

Airbase  Technologies  Division 

Tyndall  Air  Force  Base,  Florida 


Distribution  A:  Approved  for  Public  Release 


EDWARDS 

Propulsion 


Space  Vehicles 
Directed  energy 


KIRTLAND 


MESA 

Human 

Effectiveness 


BROOKS 

Human 

effectiveness 


Force  Protection  Technology 


Research  Areas 

Engineering  Mechanics 

•  Blast  Resistant  Materials 

•  Structural  Survivability 

Robotics 

•  Explosive  Ordnance  Disposal 

•  Base  Defense  &  Security 

Security  Technologies 

•  Agent  Detection  &  Characterization 

•  IED  Detection 

Reactive  Chemical  Systems 

•  Development/Demo/Testing 

•  Explosive  Operations 


Benefits  to  Warfighter 

•  Enhanced  Blast/Fragmentation  Weapon  Protection 

•  Reduced  Manpower/Time/Cost  for  Range  Clearance  Ops 

•  Increased  Safety  of  Deployed  Personnel 

•  Increased  Protection  from  Chemical  /  Biological  Threats 


BOMBOT 


Advanced  Technologies  Development 
Integrated  Base  Defense  Technologies 
Robotic  EOD  Technologies 

Automated  UXO  Response 
Technologies 

Robotics  for  Airbase  Operations  and 
Support 


Benefits  to  Warfighter 


Reduced  manpower/time/cost  for 
Range  Clearance  Ops 

Increased  safety  of  deployed 

personnel 

Technical  expertise 

Reduction  of  development  time  with 

existing  systems  and  new  capabilities 
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Test  Ranges  Approved  for  Large  Scale  Robotic  Vehicles 


Developed  &  Transitioned  Technology 


Bombot  MACE  Robo-T  rencher 


REDCAR 


ARTS 


Robotic  ATV  Carrier 
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•  Advanced  Technology  Development  expands  technology  required  for  unmanned 
systems  to  operate  autonomously 

•  Focus  is  on  technology  considered  necessary  for  an  autonomous  system 

•  New  technology  developed  or  current  technology  improved  in  a  modular  fashion 

•  Develop  enabling  robotics  technology  in  the  areas  of  Autonomous  mobility  control, 
tactical  behaviors,  world  modeling,  mission  planning,  sensor  fusion,  and  robotic 
perception 

•  Transition  technology  modules  into  applied  projects  focusing  on  specific 
applications 
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•  Develop  automated  technology  to  augment  the  security  force  mission 

•  Provide  perimeter  security  to  protect  installations  from  emerging  threats 

•  Integrate  robotic  ground,  air,  and  sea  systems  into  a  seamless  network 
with  existing  USAF  security  system  architectures  -  Integrated  Base 
Defense  Security  System  (IBDSS) 

•  Develop  object  classification  capability  to  determine  ground  traversability 

•  Develop  target  interdiction  model/planner 

•  Develop  technology  to  support  intruder  detection  on  the  move 

•  Conduct  warfighter  airfield  security  experiment 


£J1)  i 


•  Increase  EOD  operations  conducted  mostly  by  robotic  systems  under  the 
supervision  of  EOD  personnel 

•  Detection  and  neutralization  of  conventional  unexploded  military  munitions  using 
experience  based  and  adaptive  artificial  intelligence 

•  Employ  state  of  the  art  sensors  on  robotic  systems  for  detection  of  non-standard 
explosive  threats  and  neutralization  of  conventional  explosive  devices 

•  Increase  the  operational  capability  of  the  EOD  mission  personnel  by 
providing  technology  solutions  to  decrease  mission  time  and  increase 
stand-off  to  save  lives  putting  unmanned  systems  in  harm’s  way 
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•  Validate  UXO  detection  methods  using  an  unmanned  air  vehicle 

•  Unmanned  ground  vehicle  autonomously  travel  to  these  targets  for 
verification  and  elimination  utilizing  a  multi-shot  SMUD  platform 

•  Technology  will  enhance  collaborative  engagement  technology 
development  between  UAVs  and  UGVs 

•  Includes  multiple  autonomous  solutions  to  the  range  clearance  issue 

•  Develop  technology  to  support  UXO  range  clearance  operations 

•  Integrate  UXO  detection  techniques  between  a  UAV  and  UGV  platform 

•  Demonstrate  automated  detection  and  clearance  of  unexploded  items 
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•  Develop  robotic  technology  to  allow  for  unmanned  firefighting 
operations  in  hazardous  locations 

•  Increase  the  efficiency  of  ground  support  operations  through 
automation  of  the  ground  refueling  process  for  the  planned  F-35  JSF 
training  mission  at  Eglin  AFB 

•  Protect  personnel  during  transport  and  handling  of  remains  and 
prevent  the  spread  of  contamination  during  recovery  and  delivery 
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Points  of  Contact: 

Strategic  Research  Manager 
Mr.  Al  Nease 

850-283-3705  DSN  523-3705 
AI.Nease@tyndall.af.mil 

Robotics  Group  Lead 
Mr.  Walter  Waltz 
850-283-3725  DSN  523-3725 
Walt.Waltz@tyndall.af.mil 
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GSA 


U.S.  General  Services  Administration 


Federal  Acquisition  Service 


Who  are  we? 

GSA  Federal  Acquisition  Service... A  New  Service 
Representing 

General  Supplies  and  Services 
Integrated  Technology  Services 
Assisted  Acquisition  Services 
Travel,  Motor  Vehicle,  &  Card  Services 


2 


Federal  Acquisition  Service 


Today’s  Topic 

Enabling  GSA  “Tools”  (Professional  Services 
Schedule  contracts) 

GSA  e-Tools 

GSA  Support 


Federal  Acquisition  Service 


NDIA’s  Principle  Missions 

. are  to  improve  weapons  technology,  improve 

defense  management,  and  maintain  a  strong  science- 
industry-defense  team  continually  responsive  to  all 
needs  of  the  research,  development,  test  &  evaluation, 
production,  logistics  and  management  phase  of  national 
preparedness . 
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Federal  Acquisition  Service 


Conference  Objectives 

“Test  and  Evaluation  of  Autonomous  Systems 

& 

The  Role  of  the  T  &  E  Community  in  the 
Requirements  Process 


Enabling  Tools 


GSA  Professional  Services 
Schedule  Contracts 


Federal  Acquisition  Service 


Commercial  Item  Acquisition 


Expanding  the  use  of  commercial  items  in  DoD  systems  offers 
the  DoD  opportunities  for  reduced  cycle  time,  faster  insertion  of 
new  technology,  lower  life-cycle  costs,  greater  reliability  and 
availability,  and  support  from  a  more  robust  industrial  base.  It  is 
a  fact  that  for  many  technologies  that  are  critical  to  military 
systems,  the  commercial  marketplace-and  not  the  DoD-now 
drives  the  pace  of  innovation  and  development. 


DoD  Extension  to  PMBOK  ®  Guide 
June  2003 
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Federal  Acquisition  Service 


GSA  Schedules  Program 

What  is  a  Schedule? 

>  GSA  Awarded  Competitive  IDIQ  contracts 

>  Mirrors  Commercial  Buying  Practices 

>  Long-term  Contracts  Awarded  to  Multiple  Companies 
-  5  yrs.  With  Three  5-yr.  Options 

>  Forty-three  Schedules  Offer  1 1  Million-Plus  Products  &  Srv. 

>  Huge  Selection:  Over  14,000  Companies  Represented 
on  Nearly  18,000  Contracts 
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Federal  Acquisition  Service 


Multiple  Award  Schedule  Benefits 

All  Competition  Requirements  Have  Been  Met 

No  Synopsis  Required 

Prices  Have  Been  Deemed  Fair  &  Reasonable 
Terms  and  Conditions  Have  Been  Pre-negotiated 
Reduced  Need  For  Front-End  Procurement  Personnel 

Reduced  Procurement  Lead  Time 
Minimizes  Documentation  Required 
Customized  Solutions 
Direct  Relationship  With  Contractors 
No  Additional  Administrative  Fees 
Risk  of  Protest  is  Low 


Federal  Acquisition  Service 


GSA  Professional  Services 

(samples) 

Professional  Engineering  Services 

Mission  Oriented  Business  Integrated  Services 
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Federal  Acquisition  Service 


Professional  Engineering 
Services  -  Schedule  871 


Federal  Acquisition  Service 


Professional  Engineering  Services 
Scope  of  Schedule 

This  schedule  provides  a 
comprehensive  vehicle  for 
Federal  agencies  to  use 
when  obtaining  all  types 

of  engineering  services 


Federal  Acquisition  Service 


Professional  Engineering  Services 
(Schedule  871)  Industry  Partners 

Over  800  Contractors  as  of  January  2008, 
including: 

Large  Businesses 
Small  Businesses 
Disadvantaged 
HUBZone 
Veteran-owned 
SDVOB 
Women-owned 
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Federal  Acquisition  Service 


Professional  Engineering  Services-871 

Special  Item  Numbers  (SINs): 

871-1  Strategic  Planning  for  Technology  Programs/Activities 

871-2  Concept  Development  and  Requirements  Analysis 

871-3  System  Design,  Engineering  and  Integration 

871-4  Test  and  Evaluation 

871-5  Integrated  Logistics  Support 

871-6  Acquisition  and  Life  Cycle  Management 

871-7  Construction  Management 
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Federal  Acquisition  Service 


Mission  Oriented  Business  Integrated 
Services  (mobis)-874 

Special  Item  Numbers  (SINs): 

874-1  Consulting  Services 

874-2  Facilitation  Services 

874-3  Survey  Services 

874-4  Training  Services 

874-5  Support  Products 

874-6  Competitive  Sourcing  Support 

874-7  Program  Integration  and  Project  Management  Services 
874-99  Introduction  of  New  Services 
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Federal  Acquisition  Service 


GSA  Tools  for 
Test  &  Evaluation 
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Federal  Acquisition  Service 


T&E  Metrics  for  Suitability  and 
Sustainability 

SIN  871-6  Acquisition  Life  Cycle  Management  (PES) 

SIN  871-5  Integrated  Logistics  Support  (PES) 

SIN  874-7  Program  Integration  and  Project 
Management  (MOBIS) 
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Federal  Acquisition  Service 


Reducing  Total  Ownership  Costs  and 
Role  of  T&E,  SE  and  Logistics 

SIN  871-6  Acquisition  Life  Cycle  Management  (PES) 

SIN  871-2  Concept  Development  and  Requirements 
Analysis  (PES) 

SIN  871-5  Integrated  Logistics  Support  (PES) 

SIN  874-7  Program  Integration  and  Project 
Management  (MOBIS) 
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Federal  Acquisition  Service 


Test  Planning  to  Assure  Priority  for 
Assessment  of  S&S 

SIN  871-3  Systems  Design,  Engineering  and 
Integration  (PES) 

SIN  871-4  Test  and  Evaluation  (PES) 

SIN  874-1  Consulting  Services  (MOBIS) 

SIN  874-7  Program  Integration  and  Project 
Management  (MOBIS) 


Federal  Acquisition  Service 


Planning  and  Implementing 
Sustainability  as  a  KKP  Effectively 

SIN  871-3  Systems  Design,  Engineering  and 
Integration  (PES) 

SIN  871-4  Test  and  Evaluation  (PES) 

SIN  871-5  Integrated  Logistics  Support  (PES) 

SIN  874-7  Program  Integration  and  Project 
Management  (MOBIS) 
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Federal  Acquisition  Service 


Design  Techniques  such  as  Conditioned- 
Based  Maintenance  and  its  T&E 


SIN  871-3  Systems  Design,  Engineering  and 
Integration  (PES) 

SIN  871-4  Test  and  Evaluation  (PES) 


Federal  Acquisition  Service 


Test  Methodology 

SIN  871-2  Concept  Development  and  Requirements 
Analysis  (PES) 

SIN  871-3  Systems  Design,  Engineering  and 
Integration  (PES) 

SIN  871-4  Test  and  Evaluation  (PES) 

SIN  874-1  Consulting  Services  (MOBIS) 
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Federal  Acquisition  Service 


Testing  for  Realistic  Estimates  for 
Reliability 


SIN  871 -Test  and  Evaluation  (PES) 

SIN  874-7  Program  Integration  and  Project 
Management  (MOBIS) 
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Federal  Acquisition  Service 


Reducing  Total  Cost  of  Ownership 

SIN  871-1  Strategic  Planning  for  Technology 
Programs  (PES) 

SIN  871-5  Integrated  Logistics  Support  (PES) 

SIN  871-6  Acquisition  Life  Cycle  Management  (PES) 
SIN  874-1  Consulting  Services  (MOBIS) 
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Federal  Acquisition  Service 


Technologies  to  Reduce  Life 
Cycle  Cost 


SIN  871-1  Strategic  Planning  for  Technology 
Programs  (PES) 

SIN  871-2  Concept  Development  and  Requirements 
Analysis  (PES) 
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Federal  Acquisition  Service 


Field  Test  Data  and  Archiving 

SIN  871-4  Test  and  Evaluation  (PES) 

SIN  871-6  Acquisition  Life  Cycle  Management  (PES) 

SIN  874-7  Program  Integration  and  Project 
Management  (MOBIS) 
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Federal  Acquisition  Service 


Feedback  Sustainment  Lessons  to  Improve 
Requirements,  Programming  T&E  and 
Acquisition  Process 

SIN  874-1  Consulting  Services  (MOBIS) 

SIN  871-5  Integrated  Logistics  Support  (PES) 

SIN  871-6  Acquisition  Life  Cycle  Management  (PES) 

SIN  874-7  Program  Integration  and  Project 
Management  (MOBIS) 
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Federal  Acquisition  Service 


E-tools 
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Federal  Acquisition  Service 


E-Tools:  Bringing  Buyers  and  Sellers 
Together 

GSA  Advantage!® 
e-Buy 

e-Library 
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Federal  Acquisition  Service 


Market  Research  Tools 


Buyer, 


GSA 


Schedules  e-Library 

GSA  Federal  Supply  Service 


Research  vendors 


i 


GSA 


www.gsaAdvantage,  gov' 


Research  vendors 
Research  pricing 


GSA 


e-Buy 


RFQ/RFI 
Obtain  quotes 
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Federal  Acquisition  Service 


Market  Research  Web  Address 

Schedules  E-Library 
www.fss.qsa.gov/elibrarv 

GSA  Advantage!® 

www.qsaAdvantaqe.gov 

GSA  e-Buy 

access  through  GSA  Advantage!® 
www.qsaAdvantaqe.gov 


Federal  Acquisition  Service 


GSA  Support 
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Federal  Acquisition  Service 


Support  Provided  By  GSA 

Agency  outreach:  guidance,  training  &  education 

Industry  outreach:  promote,  facilitate  the  growth  of 
schedules  supplier  base 

Expedite  awards/mods  of  “potential”  suppliers 

Review  SOWs 
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THANK 

YOU! 


Frank  Parker 
(253)  931-7951 
Frank.parker@gsa.gov 


GAO 

Accountability  *  Integrity  *  Reliability 


Opportunities  to  Optimize  the 
Operational  Use  of  Unmanned  Aircraft 
Systems  and  Gain  Synergy  in 
Developing  New  Capabilities 


24th  Annual  National  Test  & 
Evaluation  Conference 
February  27,  2008 
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GAO’s  Mission  and  Related  Work 


GAO 

Accountability *  *  Integrity  *  Reliability 


•  GAO  is  the  audit,  evaluation  and  investigative  arm  of  Congress.  It 
supports  Congress  in  meeting  its  constitutional  responsibilities  and 
helps  improve  the  performance  and  accountability  of  the  federal 
government  by 

-  examining  the  use  of  public  funds; 

-  evaluating  federal  programs  and  policies;  and 

-  providing  analyses,  recommendations,  and  other  assistance  to 
help  Congress  make  informed  oversight,  policy,  and  funding 
decisions. 


•  At  Congress’s  request,  GAO  has  assessed  various  aspects  of  the 
Department  of  Defense’s  (DOD)  management  and  acquisition  of 
intelligence,  surveillance,  and  reconnaissance  (ISR)  assets, 
including  the  military  services’  unmanned  aircraft  systems  (UAS) 
programs. 
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Background 


GAO 

Accountability *  *  Integrity  *  Reliability  i 


•  Battlefield  commanders  carrying  out  ongoing  operations  are 
supported  by  DOD’s  ISR  assets,  some  of  which  are  UAS. 

•  To  meet  the  growing  demand  for  ISR  assets  at  every  level  of 
command,  DOD  is  investing  in  several  ISR  systems,  including  UAS. 

•  DOD  plans  to  invest  more  than  $21  billion  in  UAS  from  fiscal  year 
2007  through  fiscal  year  2013. 

•  Each  of  the  military  services,  as  well  as  the  U.S.  Special  Operations 
Command,  operates  UAS.  As  of  September  2007,  DOD 
components  had  more  than  5,000  unmanned  aircraft  in  inventory. 
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Background  (continued) 


GAO 

Accountability *  *  Integrity  *  Reliability  i 


•  Acquisition  of  higher  cost  and  more  complex  unmanned  systems 
raised  visibility  and  increased  congressional  scrutiny 


•  As  with  other  major  weapon  systems,  Congress  wants 

-  well-managed  acquisition  programs  meeting  cost,  schedule, 
and  performance  goals 

-  fielded  systems  providing  the  warfighter  with  the  capabilities 
needed  on  time 

-  emphasis  on  jointness,  limited  duplication,  and  strategic 
portfolio  management  of  unmanned  assets 


GAO 

Accountability  *  Integrity  *  Reliability 


Unmanned  Aircraft  Inventory  Increase,  2002- 
2007 


From  2002-2007,  DOD 
substantially  increased  its 
inventory  of  unmanned  aircraft. 


The  largest  increase  was  in  small 
unmanned  aircraft;  e.g.,  the  RQ-1 1 
Raven  and  the  RQ-1 3  Dragon 
Eye. 


Theater  and  Tactical  unmanned 
aircraft  include  the  MQ-1  Predator 
and  the  RQ-7  Shadow  200. 


Source:  GAO  analysis  of  DOD  data. 

Note:  Data  as  of  September  2007.  Small  UAS  weigh 
less  than  10  pounds,  Theater  and  Tactical  more  than 
10  pounds. 
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jk  G  A  0 

Accountability *  *  Integrity  *  Reliability 

General  Observations  on  DOD’s  Ability  to 

Optimize  the  Operational  Use  of  UAS 

•  DOD  has  experienced  a  high  level  of  mission  success  with  UAS  in 
ongoing  operations.  For  example,  UAS  have  been  used  to  identify 
time-critical  targets  and  to  strike  enemy  positions  to  defeat 
opposing  forces. 


•  While  DOD  has  taken  steps,  GAO  has  reported  that  DOD  has 
experienced  difficulties  in  the  management  of  its  UAS  programs  in 
the  following  specific  areas: 

-  Strategic  planning 

-  Operational  challenges 

-  Performance  measurement 
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Strategic  Planning 


GAO 

Accountability *  *  Integrity  *  Reliability 


•  GAO’s  prior  work  has  found  that  DOD  lacks  a  robust  oversight  framework 
and  a  strategic  plan  to  guide  UAS  development  and  investment  decisions, 
to  include  key  elements  such  as  a  clear  link  between  goals,  capabilities, 
funding  priorities,  and  needs. 

•  DOD  has  established  oversight  bodies,  such  as  the  UAS  Planning  Task 
Force  and  the  Joint  UAS  Center  of  Excellence,  to  facilitate  planning  and 
coordination.  However,  these  entities  are  advisory  bodies  and  lack 
directive  authority. 

•  In  lieu  of  establishing  an  executive  agent  for  UAS,  DOD  recently 
established  a  new  Task  Force  to  “lead  a  DOD-wide  effort  to  coordinate 
UAS  issues  and  to  develop  a  way  ahead  to  enhance  operations,  enable 
interdependencies,  and  streamline  acquisition.” 

•  At  this  point,  it  is  unclear  whether  the  Task  Force  has  the  purview  and 
authority  to  resolve  the  management  challenges  GAO  has  previously 
identified. 
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Operational  Challenges 


GAO 

Accountability *  *  Integrity  *  Reliability 


•  DOD  components  have  established  guidance  to  facilitate  the 
integration  of  UAS  into  combat  operations,  such  as  multi-service 
tactics,  techniques,  and  procedures,  along  with  a  Joint  UAS 
concept  of  operations. 

•  However,  commanders  face  challenges  in  fully  optimizing  the  use 
of  UAS  assets,  due  in  part  to  the  growing  number  of  UAS.  These 
challenges  include: 

-  Limited  interoperability 

-  Electromagnetic  spectrum  constraints 

-  Lack  of  DOD-wide  procedures  for  the  advanced  coordination  of 
UAS  into  a  theater  of  operations 
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Operational  Challenges  (continued) 


GAO 

Accountability *  *  Integrity  *  Reliability 


•  While  DOD  is  taking  several  positive  steps,  we  reported  that  the 
department’s  strategic  approach  to  managing  its  current  UAS  assets  is  not 
enabling  DOD  to  fully  optimize  its  use  of  these  assets. 

•  U.S.  Strategic  Command’s  Joint  Functional  Component  Command  for  ISR 
(JFCC-ISR)  manages  the  annual  process  for  allocating  available  ISR 
assets  to  the  combatant  commanders  to  meet  theater-level  needs, 
including  UAS.  However,  JFCC-ISR  has  limited  visibility  into  national  and 
allied  ISR  assets,  and  lacks  visibility  into  UAS  assets  embedded  and 
controlled  by  tactical  units. 

•  The  Joint  Force  Air  Component  Commander  (JFACC)  tasks  theater  UAS 
assets,  including  the  Predator  and  Global  Hawk,  against  specific  ISR 
requests.  The  JFACC  has  had  limited  visibility  into  the  use  of  UAS  ISR 
assets  controlled  by  tactical  units.  That  limitation  can  lead  to  duplicative 
taskings  as  well  as  diminish  the  commander’s  ability  to  fully  leverage  all 
available  assets. 
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Performance  Evaluation 


GAO 

Accountability  *  Integrity  *  Reliability 


•  DOD  uses  limited  quantitative  metrics  to  measure  the  effectiveness 
of  its  ISR  missions,  and  it  does  not  routinely  capture  feedback  on 
ISR  effectiveness  in  meeting  warfighters’  requirements. 


•  DOD  components,  such  as  JFCC-ISR,  have  been  tasked  with 
developing  ISR  performance  metrics.  While  they  have  made  some 
progress,  they  have  not  set  milestones  for  completing  this  effort. 


•  Without  developing  metrics  and  systematically  gathering  feedback, 
DOD  is  not  in  a  sound  position  to  validate  the  true  demand  for  its 
ISR  assets,  determine  whether  it  is  allocating  and  tasking  these 
assets  in  the  most  effective  manner,  or  acquire  new  systems  that 
best  support  its  warfighting  needs. 
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General 


GAO 

Accountability  *  Integrity  *  Reliability 


on  UAS  Acquisitions 


•  More  joint  investment  strategy  based  on  portfolio 
management  principles  needed 


•  Improved  business  case  development  and  lower 
risk  acquisition  strategies 


•  More  effective  systems  engineering  practices  need 
to  be  followed 


•  Better  cost  and  schedule  outcomes 
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GAO 

Accountability  *  Integrity  *  Reliability 


Opportunities  Exist  For  Greater  Collaboration 
Across  UAS  Programs 


•  Greater  collaboration  on  similar  programs  but  different 
services  would  create  a  more  efficient  and  affordable 
approach 


•  Army  and  Navy’s  collaboration  on  Fire  Scout  has 
achieved  benefits 


•  BAMS  could  benefit  from  greater  collaboration 


•  Warrior  and  Predator  have  been  slow  to  exploit 
synergies 
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UAS  Are  Not  Immune  to  Common 
Development  Problems 

•  Immature  technologies 


GAO 

Accountability  *  Integrity  *  Reliability 


•  Requirement  changes 


•  Design  changes 

•  Quality  and  reliability  problems 
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GAO 

Accountability  *  Integrity  *  Reliability 


Examples  of  UAS  Programs,  Problems 
Encountered,  and  Impact 


System 

Problem  Encountered 

Impact 

Global  Hawk 

•Concurrent  acquisition  strategy 
•Immature  technology 
•Requirements  changes 
•Design  changes 

•Development  cost  growth  of  261  percent 
•Schedule  delay  of  36  months 
•Several  program  restructures 
•Increased  investment  in  legacy  systems 

MQ-9  Reaper 

•Concurrent  acquisition  strategy 
•Immature  technology 

•  Development  cost  growth  of  13  percent 

•  Schedule  delay  of  7  months 

Warrior 

•Concurrent  acquisition  strategy 
•Immature  technology 

•Development  cost  growth  of  21  percent 
•Schedule  delay  of  9  months 

Broad  Area  Maritime 
Surveillance 

•Immature  technology 
•Funding  availability 

•Schedule  delayed  39  months 
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GAO 

Accountability  *  Integrity  *  Reliability 


DOD  Acquisition  Process  and  Best  Practices 


Concept 
Decision 


Program 

Start 


Production 

Commitment 

XcX 


Technology 

“  Production  & 

Operations  & 

Development 

Deployment 

Support 

Design  Readiness 

FRP  Decision 

Review 

Review 

Technology 

Development 


Knowledge  Point  1 

Technologies  and 
resources  match 
requirements 


Product  Deve 

Integration 

opment 

Demonstration 

Production 

Program 

Launch 


Knowledge  Point  2 

Design  performs  as 
expected 


Knowledge  Point  3 

Production  can  meet 
cost,  schedule,  and 
quality  targets 


Key  pre-system  development  markers: 

--Integrated  portfolio  management  strategy 
--Systems  Engineering/Requirements  Analysis 
--Technology  maturation  and  transition 


Key  development  markers: 

--Sound,  executable  business  case 
--Evolutionary  approach 
--Knowledge  based  decision  points 
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GAO 

Accountability  *  Integrity  *  Reliability 


Comparison  of  Predator  B  and  Global  Hawk 
Acquisition  Plans 


Predator  B 


Product  Development  Start 
Production  Start 


Initial  Operational  Test  and  Evaluation 
Increment  1  - 


Product  Development  Start 


Production  Start 


Increment  2 


Global  Hawk 

Product  Development  Start 

Production  Start 


Initial  Operational  Test  and  Evaluation 


Technology  Development 


System  Development 


Production 
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GAO 

Accountability  *  Integrity  *  Reliability 


Example  of  Global  Hawk  Unit  Cost  Growth 


ACTD 

< - > 


Acquisition  program 

< - > 


■  Unit  recurring  flyaway  costs 
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Related  GAO  Products 


GAO 

Accountability *  *  Integrity  *  Reliability 


•  Unmanned  Aircraft  Systems:  Advanced  Coordination  and  Increased 
Visibility  Needed  to  Optimize  Capabilities ,  GAO-07-836  (Washington,  D.C.: 

•  Intelligenc 
on  DOD’s 
Assets,  ar 
19,  20(37). 

•  Defense  Acquisitions:  Greater  Synergies  Possible  for  DOD’s  Intelligence, 
Surveillance,  and  Reconnaissance  Systems,  GAO-07-578  (Washington, 
D.C.:  May  17,  2007). 

•  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  G AO- 
07-406SP  (Washington,  D.C.:  March  30,  2007). 

•  Best  Practices:  An  Integrated  Portfolio  Management  Approach  to  Weapon 
System  Investments  Could  Improve  DOD’s  Acquisition  Outcomes,  GAO- 
07-388  (Washington,  D.C.:  March  30,  2007). 

•  Unmanned  Aircraft  Systems:  Improved  Planning  and  Acquisition  Strategies 
Can  Help  Address  Operational  Challenges,  GAO-06-61 0T  (Washington, 
D.C.:  April  6,  2006). 


e.  Surveillance,  and  Reconnaissance:  Preliminary  Observations 
Approach  to  Managing  Requirements  for  New  Systems,  Existing 
id  Systems  Development,  GAO-07-596T  (Washington,  D.C.:  April 
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Related  GAO  Products  (continued) 


GAO 

Accountability *  *  Integrity  *  Reliability 


•  Unmanned  Aircraft  Systems:  New  DOD  Programs  Can  Learn  from  Past 
Efforts  to  Craft  Better  and  Less  Risky  Acquisition  Strategies,  GAO-06-447 
(Washington,  D.C.:  March  15,  2006). 

•  Unmanned  Aircraft  Systems:  Global  Hawk  Cost  Increase  Understated  in 
Nunn-McCurdy  Report,  GAO-06-222R  (Washington,  D.C.:  December  15, 
2005). 

•  Unmanned  Aircraft  Systems:  DOD  Needs  to  More  Effectively  Promote 
Interoperability  and  Improve  Performance  Assessments ,  GAO-06-49 
(Washington,  D.C.:  December  13,  2005). 

•  Unmanned  Aerial  Vehicles:  Improved  Strategic  and  Acquisition  Planning 
Can  Help  Address  Emerging  Challenges ,  GAO-05-395T  (Washington, 
D.C.:  March  9,  2005). 

•  Unmanned  Aircraft  Systems:  Changes  in  Global  Hawk’s  Acquisition 
Strategy  Are  Needed  to  Reduce  Program  Risks,  GAO-05-6  (Washington, 
D.C.:  November5,  2004). 
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GAO  Contact  Information 


GAO 

Accountability  *  Integrity  *  Reliability 


Sharon  L.  Pickup 

Director,  Defense  Capabilities  and  Management 
U.S.  Government  Accountability  Office 
pickups@gao.gov  or  (202)  51 2-961 9 

Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 

sullivanm@gao.gov  or  (202)  512-4841 
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NDIA 

National  Test  &  Evaluation 

Conference 


Unmanned  Vehicle  Synthetic 

Environment 


February  26, 2008 

Angelo  Prevete 
Program  Manager 
Lockheed  Martin 
Simulation,  Training  &  Support 

Tel:  407-306-5019 


Myron  Mills 
Project  Manager 
Lockheed  Martin 
Missiles  &  Fire  Control 
Combat  Maneuver  Systems 

Tel:  972-603-7786 


Introduction 


•  Description  of  an  Unmanned  Synthetic  Environment 

-  Operational  vs.  Simulation  Architecture 

•  Example  Control  Elements 

•  Video  Examples  of  Real-time  Sensor  Images 

•  When  a  UxV  Synthetic  Environment  (SE)  could  be  used 

•  Corporate  Research  and  Development  Efforts 

-  Unmanned  Ground  Vehicle  Technology  Transformation  Program 
(TTP) 

-  Fast  Inshore  Attack  Craft  (TTP) 

•  Conclusion 


Lockheed  Martin 


S — ! 


UxV  Architectures 

UxV  Live/Operational  Environment 


Simulated  Terrain/Atmospheric  Imagery  (e.g.  Visual,  IR,  SAR) 


Lockheed  Martin 


Example  Control  Element: 

Universal  Controller  Prototype  (UCP) 


CIB  1  meter  WGS84 


N  34.59216°  E  043.68073° 


Select 
Chan.  2 


WptPlan 

/Follow 


Camera 


/Sensor 


Selected  VIP 


Vehicle  Type 


Mode 


Select 


Adjust 

Sensor 


Manual 


Sensorl 


UXV1 :  Sending  START  ENGINE  command. 
UXV1:  Sensorl  selected. 

UXV1 :  Sending  TURN  SENSOR  ON  command. 
UXV1 :  Sensor  2  selected. 

UXV1 :  Sending  TURN  SENSOR  ON  command. 
CAUTION:  Handgrips  Activated! 


MPH/MS  Hdng 


Position 


Turn  On 
Sensor 


Set  Mast 
Height 


PackBot 

Video 


34.592784  43.688865 


V/R 

Control 


Association  List 


start 


O  FalconView  -  LIMITED. 


HRI 

Page 

Config. 

Page 

Params 

Page 

Routes 

Page 

Wpts 

Page 

Threats 

Page 

ANN 

Page 

Vehicle 

State 

Plan 

Status 

INS 

Display 

HRI 

GPS 

Camera 

Pose 

V/R 

Display 

SW 

Versions 

Sensor 

Info. 

Weapon 

Info. 

Log 

Events 

Misc 

<1 

D> 

ESTOP 

Stop 

Engine 

Start 

Engine 

Disable 
Cm  d/Ctrl 

Assoc. 

Assoc. 

UXV 

fill  00:49:35.00 

U 

& 

|  File  Player  Help _ 

Lockheed  Martin 


Example  Control  Element: 

Generic  Unmanned-Vehicle  Supervisory  Segment  (GUSS) 


v 


y  ANLAS  Job  Bar 


-!~1  x 


Configuration  System  UAV  Window  Help 

ion  Control  UAV-Sensor  Control  Prompts 


I  0  UAV  Mission  Control 


File  View  Map  Plot  Controls  Bookmarks  Tools  TDAs  Tracks 


Ownship 


-=lSl-Xj 

Mission  Control: 


1  ^  UAV  108  -  Status  Overlay 

x 

Mission  Time  Remaining 

1:54:46 

Reserved  Recovery  Time: 

0:10:54 

Current  Action  Time-To-G 

o:  M/A 

Current  UAV  Action: 

Orbiting  -  Idle 

UAV  Location: 

34:37:1  7N  43:40:32E 

FOV  Slant  Range  /  Zoom: 

2548m  l  75% 

UAV  Comms  Status: 

UAV  Altitude  (AGUMSL): 

UAV  Velocity  (Air/Grnd): 

8234ft  /  8600ft 

75kt  /  75kt 

|  #  Prompts 

-ISJ  x| 

UAV  229  Transit  Complete,  C 
UAV  229  beginning  Point  Sei 

Orbiting  Idle. 

arch 

I 

,1 

When  a  UxV  Synthetic  Environment  (SE)  could  be  Used 


Lockheed  Martin 
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Corporate  IR&Ds 

Unmanned  Ground  Vehicle  Technology  Transformation 
Program  (TTP) 

Objectives: 

-  Develop  an  Unmanned  Vehicle  (UxV)  Simulation  Test-bed  to  evaluate  the  impact  of 
unmanned  systems  on  combat  effectiveness,  lethality  and  survivability  of  the  platoon 

-  Provide  Tool  for  deriving  unmanned  systems  requirements,  CONOPS,  and  Tactics 
Techniques  and  Procedures 

-  Simulate  a  Light  Infantry  Platoon  conducting  operations  in  complex  urban  scenarios 

Key  ideas  to  examine: 

-  “Rules”/Logic  For  Multi-vehicle/Multi-controller  Use 

-  Autonomy,  Situation  Awareness,  Collaboration  &  Role  De-confliction 

-  Control  Hand-off  At  The  Unit  Level 

-  Element  “Out  Of  Fight”  Taking  control  Of  Unmanned  Systems 

Synthetic  Environment  Includes: 

-  4  Simulated  Video  Channels:  Multi-vehicle,  Dynamically  Selectable 

-  “Build-a-Bot”,  Scalable,  Unmanned  Vehicle  Simulation  (UGV  and  VTOL  UAV) 

-  Universal  Controller  Prototype  with  simultaneous  control  of  multiple  types  of  UxVs 

-  Joint  Semi-Automated  Forces  (JSAF)  for  Manned  Ground  Vehicle  and  Dismounted 

Infantry  Simulation  s 


Lockheed  Martin 


h  n-T  TT 

In  11 1 11 

Bg"l  il  1 

Unmanned  Vehicle  (UxV)  Simulation  Test-bed 

Controller  and  Stealth  Displays 


Lockheed  Martin 


II 

1 

T 

V  5\  * 

’■'■‘V’SI 

Vl  ,  *  \(J 

1 

1  j  II 

II 

1  r  a 

T  1  ► - 1 

Unmanned  Vehicles  (UxV)  Simulation  Test-bed 

Live  Vehicle  Static  Displays 


Lockheed  Martin 
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Unmanned  Vehicle  (UxV)  Simulation  Test-bed 

Live  Vehicle  Static  Displays 


Lockheed  Martin 
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Unmanned  Vehicle  (UxV)  Simulation  Test-bed 

Customer  Demonstrations 


Lockheed  Martin 


Corporate  IR&Ds  (Cont) 

Fast  Inshore  Attack  Craft  (FIAC)  TTP 

•  Experimental  platform  used  to  demonstrate  ship  protection  within  5 
NM  radius 

•  Uses  LM  Sea  Slice  vessel,  Hellfire  Missiles,  Radar,  EO/IR  sensors 
and  UAVs 

•  Live  and  simulated  experiment  performed  on  Sea  Slice  vessel 
operating  in  vicinity  of  Channel  Islands  and  San  Diego 

•  Synthetic  Environment  includes: 

•  Embedded  simulation  capability  -  10  simulated  video  channels 

•  Santa  Cruz  /  Santa  Rosa  Channel  Islands  /  San  Diego  coastline  DBs  for  exercise 
area 

•  Radar  Track  and  sensor  simulation  provided  to  onboard  C4I  devices 

•  Hellfire  simulation 

•  Joint  Semi- Automated  Forces  (JSAF)  used  to  generate  naval  insurgent  attack 
scenarios 


Lockheed  Martin 


Operational  Simulation  &  Visualization 


Simulation  Capabilities 

•  Full  Fidelity  Synthetic  Environment 

•  Natural  representation  of  geo-specific  operating  areas 

•  Robust  Semi-Automated  Computer  Generated  Forces  (CGF) 

•  Hostile  target  boats,  friendly  and  neutral  clutter 

•  Scenario  Generation  functionality 

•  Real-time  Sensor  Simulation 

•  EO,  IR,  and  Radar  simulation  at  Combat  Management  Operator 
stations 

•  Realistic  Weapons  Engagement 

•  Hellfi re  fly-out  and  engagement  of  CGF 

•  Embedded  in  Tactical  Systems 

•  Distributed  environment  driving  situational  awareness  at  each 
operator  station 


SRC:  DHWK-2  DTG:  1 621 33:38  Z  May  06 


Operational  Value 

•  Initial  Team  Training  through  Mission  Rehearsal 

•  Concept  Development  and  Experimentation 

•  Tactics  and  Doctrine  Development 


Lockheed  Martin 


Sea  Slice  Dockside 


Sea  Slice  Mast  Sensors 


Simulated  Attack 


Lockheed  Martin 
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Hellfire 


Lockl 


.aunch  Pedestal 


Conclusion 


•  Transparency  in  HMI  operations  whether  performing  live  or  in 
simulation  is  the  goal 

•  Embedded  simulation  provides  extremely  cost  effective  method  to 
maintain  team  readiness 

•  Required  only  15  minutes  on  Sea  Slice  to  transition  from  simulated  to 
live  operations 

•  Enables  disparate  C4I  devices  to  operate  collectively  on  same  Training 
and  Mission  Rehearsal  scenarios 

•  A  representative  suite  of  equipment  incorporating  the  described 
capabilities  has  been  brought  to  the  conference.  The  UxV 
configuration  is  available  for  demonstration  and  further  discussion  in 
the  NDIA  Display  area.  Please  feel  free  to  stop  by. 


Lockheed  Martin 


Joint  Ground  Robotics  Enterprise 


Ground  Robotics  Test  and 

Evaluation: 

Are  We  Ready? 

Presented  at 

National  Industrial  Defense  Association 

27  February  2008 
Mrs.  Ellen  M.  Purdy 

Enterprise  Director,  Joint  Ground  Robotics 
OUSD(ATL)/PSA/LW&M 
ellen.purdy@osd.mil 
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Agenda 


•  If  Past  is  Prologue . Then  Robotics  are 

‘Coming  to  a  Theater  Near  You’ 

•  Where  are  We  Headed?  (and  Are  We 
Ready?) 

•  Thoughts  on  What  is  Needed  to  be  Ready  to 
Test 

•  Hopeful  Signs  of  Progress 

•  Conclusions 
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Ground  Robots 
Prov[ng  Their  Worth... 


#****++ 
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Progress  in  Autonomy 
and  Cognjtion 


Million 
instructions / 
second 

J 

IB— 


10M — 


100K _ 


Key  Challenges 

•  High  speed  mobility 

-  Improved  perception 

-  Reliable  data  fusion 

•  Safe  operations 

-  Detect  &  classify  stationary 
people  &  objects 

-  Collision  avoidance 

•  Design  for  tactical  behavior 
flexibility 

-  Understand  environments 

-  Decision  processing 

-  React  appropriately  to 
contact 

•  Collaboration 

-  Conduct  collaborative 
missions  with  mixed 
manned/unmanned  force 

-  Collaborative  air-ground 
operations 


Air/Ground 

Collaboration 


Demo  Ilia 


True  follower— follows  in 
tracks  of  manned  leader 


Crusher/DARPA 


On-road 

convoy  operations 


Near  autonomous 
team  members 


Armed  Reconnaissance 
Vehicle— RSTA 


1999 


2005 


2008 


2011 


2015... 
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Ground  Robotics  are  Saving  Lives, 
But  How  Do  We  Test  to  Avoid: 


•  The  first  robot  to  injure  a 
human  in  testing? 


*  The  first  robot  to  be  captured 
and  turned  on  US  troops? 


•  The  first  robotic  friendly  fire 
incident? 
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What’s  on  the  Horizon? 


•  Recon  in  severely  restricted 
terrain 


•  Future  software  will  allow  the 
Snakebot  to  learn  on  its  own  by 
experience 


Snakebot 


•  Provides  the  ability  to  navigate 
over  rough,  steep  terrain  where  a 
wheeled  robotic 

vehicle  would  likely  get  stuck  or 
topple  over 
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What’s  on  the  Horizon? 


Battlefield  Extraction-Assist  Robot 


Currently  in  the  proof-of-concept 
development  phase  for  US  Army’s 
Telemedicine  and  Advanced 
technology  Research  Center 

Designed  to  find,  pick  up  and  rescue 
people  without  risking  additional 
human  life 

Upper  body  controlled  by  hydraulics 

A  mobility  platform  that  features  two 
independent  sets  of  tracked  “legs” 

Features  dynamic  balancing 
behavior  (DBB)  while  on  its  “ankles”, 
“knees”  or  “hips”  7 


What’s  on  the  Horizon? 


Little  Dog 


Big  Dog 


•  Developed  under  the  Defense  Advanced 
Research  Projects  Agency’s  (DARPA) 
Learning  Locomotion  program 


*  Expected  Locomotion  Strategy: 

*  Develop  a  library  of  moves  to  traverse 
terrain  elements 

*  Recognize  similar,  already  learned 
elements  and  modify  as  required  in  real 
time 

*  Best  results  will  be  ported  to 
Big  Dog 


•  Goal  is  to  learn  how  to  traverse  large, 
irregular  obstacles  with  a  high  degree  of 
freedom  robot 
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What’s  on  the  Horizon? 


*  Developed  by  Carnegie  Mellon 

University  to  assess  the  Crusher 

capabilities  of  large,  unmanned 
ground  vehicles  operating 
autonomously  in  a  wide-range 
of  complex,  off-road  terrains 

*  Made  of  high-strength 
aluminum  and  titanium  to 
withstand  below-hull  strikes 
from  boulders  and  tree  stumps, 
and  a  nose  designed  to  absorb 
the  impact  of  major  collisions. 


Lessons  from  SWORDS 
Safety  Testing? 


•  The  SWORDS  Spiral  1  was  subjected  to  testing  for  a  total  of 
1,111  operational  hours  at  Aberdeen  Proving  Ground 

•  The  following  design  changes  were  made  solely  to  increase 
safety.  In  most  cases  performance  is  slightly  degraded  due 
to  the  design  change.  All  design  changes  were  included  in 
the  final  System  Safety  testing. 

•  Increased  number  of  steps  in  firing  sequence 

•  Limit  operation  of  the  system  to  ranges  where  the  operator  can  see  the 
vehicle  line  of  sight 

•  Loss  and  regain  of  communications  between  the  OCU  and  vehicle  will 
cause  the  operator  to  re-run  arming  and  firing  sequence,  regardless  of 
ready  state  prior  to  communications  loss 

•  Weapon  cannot  be  fired  while  platform  is  moving  (Weapon  firing  is 
inhibited  while  joystick  is  engaged  to  move  vehicle)  » 

•  Vehicle  “Kill  Switch”  added 
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Thoughts  on  What  May  Be 
Needed  to  Test  Successfully. 


•Performance  adjudication:  wheels,  tracks,  legs, 
snake  motion... which  one  more  operationally 
effective? 

•Definition  of  levels  of  autonomy  and  standard 
performance  metrics  for  each... how  do  you 
verify  that  you  have  achieved  the  desired  level 
of  autonomy? 
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Thoughts  on  What  May  Be 
Needed  to  Test  Successfully. 


*  Safety  Releases-  Are  we  ready  for  armed 
autonomous  robots  (e.g.  SWORDs  limitations)? 

•  The  TRUST  Factor-  If  humans  do  not  trust  robot  to 
perform,  then  inherent  capability  may  not  be 
achieved. ..how  do  we  measure? 
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Test  Standards  and 
Instrumentation 


•  Standard  Test  Methods,  Metrics 


•  E-Stops,  data  logging,  position 

•  How  much  latency  is  too  much? 


VICTIM  BOXES  WITH  HOLES 


ORANGE  t 

Iff 


YELLOW  ARENA  jpf 

RANDOM  MAZE 

PITCH  &  ROLL  RAMP  FLOORING  (10*  ) 
DIRECTIONAL  VIC  TIM  BOXES 
(FOR  AUTONOMOUS  ROBOTl^ 


ONFINED  SPACES  (UN 


FULL  CUBIC  STEPFIELDS 
STAIRS  (40*  ,  20CM  RISERS) 
RAMP  (45*  WITH  CARPET) 
PIPE  STEPS  (20CM) 
DIRECTIONAL  VICTIM  BOXES 
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Effective  Tests 


Computer  vision  may  not  be  as  good  as  thought, 
according  to  MIT  study 

Cathryn  M.  Delude,  McGovern  Institute 
January  24,  2008 

•  Apparent  success  may  be  misleading  because  the 
tests  being  used  are  inadvertently  stacked  in  favor  of 
computers 


•  Caltech  101  database,  intended  to  test  computer 
vision  algorithms  against  the  variety  of  images  seen 
in  the  real  world 

•  CaltechlOl  'natural'  images  fail  to  adequately 
capture  real-world  variability 


The  human  brain  easily  recognizes  that  these 
cars  are  all  the  same  object,  but  the  variations 
in  the  car's  size,  orientation  and  position  are  a 
challenge  for  computer-vision  algorithms. 
Image  /  Nicolas  Pinto 


—  \  /  % 


i  s  \  <*» 


14 


One  Possible  Answer 


•  Provides  controlled  data  for 
autonomous  navigation 
algorithm  development 

•  Provides  a  systematic 
process  to  investigate  fusion 
of  local  and  global  mobility 
data 

•  Brings  an  understanding  of 
how  complex  environments 
effect  UGV  autonomous 
operations 

•  Mission  planning  functions 
based  on  sensor  perceptions 


Virtual  Autonomous  Navigation  Environment  (VANE) 
-Testbed  Process 


Vehicle  performance 


Simulated  Terrain  Physics 


* 

Global  vehicle 
performance  maps 
from  standard  digital 
data 


Soil  strength 
Vegetation  densify"" 
Longitudinal  force 
Lateral  force 
T  raction 
Resistance 


Local  vehicle  performance 
maps  from  sensor  data 


/ 


Fusion  of  Local  and 


Global  mobility- 
critical  data  to 


determine  UGV 
mobility  course  of 
action 


Conclusions 


*  It  is  not  a  matter  of  ‘If,  but  ‘When’ 


•  The  Test  Community  has  work  to  do  to  ensure 
we  are  ready  for  the  challenges... 

•  TOPS 

•  Metrics 

•  Infrastructure 

•  Modeling  &  Simulation 


One  Additional  Thought... 

Robots  are  coming  to  the  fight... are  we  capitalizing  on 
them  in  supporting  our  testing,  e.g.  targets,  nodes  on 
the  network,  etc... 
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NDIA  Autonomous  Vehicle 
Test  and  Evaluation  Conference 


Semi-Autonomous  Dispenser  Transport  Vehicle 

for  Undersea  Sensors 


System  Integration  Test  Results  and  Lessons  Learned 


25  -  28  February  2008 


CDR  Joseph  Conway  USN  (Ret.) 
Director,  Systems  Engineering 
AMRON 


LCDR  Pete  Reinagel  USN  (Ret.) 
Test  &  Evaluation,  Assistant  PM 
SYS  Technologies 


Public  Release.  Statement  A.  Distribution  is  Unlimited. 
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Outline 


*  Introduction 

*  Background 

'Challenges  of  Delivering  Undersea  Sensors  via  LCS 

*  Development  and  Characteristics  of  Dispenser 
Transport  Vehicle  (DTV) 

'High  Level  Platform  and  Design  Constraints 

'Test  and  Evaluation 

*  Lessons  Learned  and  Recommendations 


UNCLASSIFIED 
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Advanced  Deployable  System  (ADS) 

Program  Overview 


*The  primary  ADS  function  is  to  provide  undersea 
surveillance  in  shallow  littoral  waters 


-ACAT  1C  POR 
-  IOC  in  2009 


-  Now  cancelled 

•  ADS  program  developed  multiple  semi-autonomous 
delivery  vehicles,  known  as  Dispenser  Transport 
Vehicles  (DTV)  to  install  undersea  surveillance  sensors 

-Lockheed  Martin,  Riviera  Beach  built  8  DTVs  in  2007 

•  4  for  developmental  testing 

•  4  for  operational  testing 


UNCLASSIFIED 
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ADS  Subsystem  Overview 


•  Purpose:  Demonstrate  undersea  surveillance  system  for  use  against  enemy 
submarines  in  a  littoral  environment 


Array  Installation 
Module  (AIM) 

•  Cable  pack 

•  Battery/ 
Electronics 

•  DTV 


•  Buoy  and  Anchor 


Tactical  Interface 
Subsystem  (TIS) 
Buoy 


ADS  buoy 


Mid-water 
floats  &  riser 


Sensor  Subsystem  (SS) 

•  4  sensor  subsystems  (arrays) 

•  1  TIS  buoy 


Analysis  &  Reporting 
Subsystem  (ARS) 

On-Board  ICP-based  Processing 
Reporting  using  ship  Comms 


Installation  Support 
Subsystem  (ISS) 

•  Handling  &  stowage  gear 

•  Drops  AIM  astern  at  speed 


rF  connection 


Inter-AIM  Cable 


Array 


Trunk  Cable 


Anchor 


At  Sea  Demonstration  of  Deployable  Littoral  USW  System  -  Nov  07 
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ADS  System  Requirements 


'Detect  submarines  and  surface  contacts  in  the  littorals 
with  ability  to  provide  persistent  wide  area  surveillance 


-  Pre-processing  in  the  buoy  to  reduce  bandwidth 

-Contact  ID,  classification,  localization,  and  tracking  done 
by  Fleet  STGs  on  Littoral  Combat  Ship  (LCS) 

*  Deployment  from  LCS  -  1st  increment 


*  Clandestine  delivery  -  2nd  increment 
'Array  deployment:  Semi-autonomous  from  DTV 


•Pd:  0.8 -0.9 


*  Installation  time:  4-8  hours  /  string  of  4  arrays  +  1  buoy 

*  Installation  depth  requirement  classified 


UNCLASSIFIED 
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ADS  on  LCS 
CONOPS  Overview 


•  ADS  Mission  Module  stored  in  forward  deployed  area 
'Module  loaded  when  ordered 

-Mission  Planning  was  aided  by  COMUNDERSEASURV 

-The  LCS  would  rapidly  go  to  the  Operating  Area  and 
install  the  sensor  portion  of  ADS 

•  Rapid  installation  possible  using  an  AIM  -  DTV  for 
array  placement 

-The  LCS  with  the  Analysis  and  Reporting  Subsystem 

•  monitored  Contacts  of  Interest  and 


•  reported  to  higher  authority 

-  LCS  could  be  many  miles  from  the  sensors  and  up  to  45 
miles  from  the  Comms  Buoy 


The  DTV  is  vital  to  making  ADS  operational. 
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LCS  Program  Overview 


*Sea  Frame  handles  Mission  Modules 
*  Modules  transported  to  create  Mission  Packages 
'Modules  developed  to  an  Interface  Control  Document 
'Floating  baseline  (hulls  differ,  storage  differ,  handling) 


GENERAL  DYNAMIC 

Littoral  Combat  Ship 


(S ENE RAESDYNAMICS  Ssth  tron  Works 
EWERAL  DYNAMICS  Advanced  Information  Systems 
mtal  USA  SYSTEMS  North  America 

Maritime  Applied  Physics  Corporation  ,  CAE 


— 


Two  Sea  Frames  increased  ADS  Design  Efforts  and  Risk 
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Concurrent  Spiral  Development 

Touch  Points 
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AIM-LCS  Interface  Development 


*  The  Array  Interface  Module  (AIM)  is  the  LCS  Interface 

-  Design  Constraints 

•  Mission  Bay  access  and  space 

•  Ability  to  maneuver  /  handle  ADS  equipment  from  all 
stowage  locations  using  shipboard  transport  equipment 

•Topside  and  Mission  Bay  environmental  conditions  (EMI) 

•  Deck  Strength 

•  Electrical,  safety  (WSESRB) 

•  Size,  weight,  power  of  AIM  impact  on  DTV 

•  Speed,  sea  state,  height  at  drop,  ice,  etc. 

•  Staging  of  DTVs  versus  NAVSEA  901 B  Shock 

requirements _ 

Vehicle  Design  must  consider  the  Host  Vessel 
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Key  Design  Characteristics 


AIM  has  to  safely  transcend  to  the  ocean  floor 

-Withstanding  Shock,  Vibration,  and  Slope 

Array  deployment  from  DTV 

-Array  coiled  inside 

-Pulled  out  of  DTV 
as  it  swims  away 


Dispenser  Transport 
Vehicle  (DTV) 

•Array  and  dispenser 


Array  Installation 
Module  (AIM) 
•Inter-array  Cable  Pack 
•Electronics  Node 
•DTV 
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Array  Installation  Module 


UNCLASSIFIED 
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Vehicle  Overview 


Acoustics 

Array  Dispenser  Compass  Transponder) 


Nose  Foam 


Altimeter 


Interface  Ring 


Board 

o  .u,  ^  IMU 

Bulkhead 


Electronics  Tray 


VB  Assembly 


Energy  Assembly 


Tailhook 


DVL 


Aft  Shroud  &  Hub 


Impeller 


Forward  Shroud 


Motor  Assemfcfy 


STATE;  MASTER_E>P 
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DTV  Configuration 


Tjjfiimv 


Fwd  Variable 
Ballast 


Aft  Variable  IMU 
Ballast 


Shaft 


Fins 


Frangibolt 


Array 
Dispense 


Section 

Seal 


Spacer 
Propulsor 
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Dispenser  Transport  Vehicle 
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Key  Developmental  Tests 


*  Build-up/risk  reduction 

-Modeling  &  simulation,  analysis,  and  bench  testing 

*  Ramp  test  of  DTV 

-  Understand  shock  load  and  interface 

*  Drop  test 

-Verify  acceptable  shock  load 

*  Deployment  from  AIM 

-  Multiple  open  water  tests 

*  Control:  modeling  /  at  sea  demo  /  re-design 

-  Invaluable  contribution  of  Mr.  William  Zirke  of  Penn  State 
•  Fin  spacer  and  Fin  Hub  to  reduce  swirl 
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System  Integration  Test  (SIT) 

Overview 


SIT  used:  a  single 
ADS  string: 

•  4  AIMS 

•  trunk  cable 

•  anchor  /  buoy 

Acoustic  data  received 
from  the  string  was 
transmitted,  via  buoy 
radio  to  ADS  radio,  for 
ARS  processing 
aboard  the  USNS 
SIOUX. 
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System  Integration  Test  (SIT) 


•Objective: 

-“Install  an  ADS  Array  (straight)  with  AIM/DTV” 


-The  variance  on  “straight”  impacts  localization  accuracy 

•6-13  November  2007 
•Southern  California 
•  Sea  States  1-5 


Major  System  Test  Effort 
•  62  Test  Personnel  at  Sea 
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SIT  Results 


*  DTV  anomalies  -  only  1  of  4  deployed  the  array  properly 

•  Key  Events 

-  Initial  indications  suggested  all  4  arrays  were  successfully 
installed 

-  After  the  ICP  was  initialized,  determined  the  arrays  did  not 
deploy  correctly  on  AIMs  1  through  3 

•  AIM  4  electronics  bottle  later  failed 


-  Remote  Operated  Vehicle  (ROV)  used  extensively 

•  Bypassed  AIM  4  by  splicing  cable  from  Node  #3  to  the  buoy 


•  Video  confirmed  the  DTVs  1-3  failed  to  deploy  arrays;  each 
left  the  AIM,  but  only  deployed  a  portion  of  the  leader  cable 

•  Attached  line  to  DTV  so  ship  could  pull  array  out  (#1  and  #2) 


-  Manual  deployment  too  late  for  SSN  tracking  (schedule  issue); 
EMATT  was  used  instead  ■still  Met  83%  of  Test  Objectives  at  SIT 
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DTV  Post-SIT  Study 


0 


Depth 

(ft) 


1000 


1500 


2000 
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ROV  Video  of  DTV  Operations 


DTV  on  bottom  after  deployment 
with  leader  cable  extending  behind 


88:31:49 


811  0? 


11/13/0? 


DTV  being  pulled  by  ship 
to  deploy  the  array 
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Lessons  Learned 
from  DTV  Testing 


1 .  Vehicle  control  issues 

-  Must  have  sufficient  Flow  over  Control  Surfaces 

-  Modeling  did  not  accurately  predict  actual  flow  behavior 

2.  Snood  foam  Compressed  at  max  depth 

-  Design  margin  for  Worst  Case 

3.  Release  method  needs  to  be  as  simple  as  possible 

-  Frangible  bolts  used;  design  issues  /  complications  caused  test  failures 

-  Issues  masked  other  problems  but  schedule  constraints  precluded 
rerunning  the  test 

4.  Insist  on  a  Full  Deployment  Test  at  max  depth 

-  Risk  is  significantly  higher  by  not  testing  at  max  depth 

5.  Verify  pressure  ratings  of  components  that  are  subject  to  sea  pressure 

-  Make  this  a  Critical  Design  Review  focus  item 

6.  Variable  ballast  should  not  have  been  Timer  Controlled 

Only  works  if  everything  else  is  working;  otherwise  causes  system  failure 
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Recommendations 


•  To  re-use  the  DTV,  we  would: 

1 .  Upgrade  the  foam 

•  Ensure  all  components  can  withstand  sea  pressure 

2.  Strengthen  the  transom  plate 

•  Verify  components  won’t  bind  under  pressure 

3.  Design  out  the  Variable  Ballast  Control  Timer 

•  Alter  the  DTV  so  the  ballast  varies  as  array  cable  is 
paid  out 

4.  Reduce  shock  load  on  the  AIM  /  Cable  Pack 

•  Consider  the  environment 

5.  Design  out  the  frangible  bolts 

•  Keep  it  simple! 
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Summary 


*The  Dispenser  Transport  Vehicle  is  a  valid  concept 

*  Remote  installation  is  still  highly  desired  at  some  sites 
and  the  DTV  can  be  an  enabler 

-  Some  Design  modifications  and  more  testing  is  required 
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DTV  Material 


Foam  Nose 


Fiberglass 

Dispenser 

Aluminum 

Bulkhead 

Aluminum 
Cast  Hull 

/ 

Urethane 

Shroud^ 

Aluminum 

Cast.Hub 


UNCLASSIFIED 


25 


UNCLASSIFIED 


Vehicle  Overview 


Variable  Ballast  System 


FWD  VB 

Aft  VB 

Tank 

Tank 

Propulsor 


Impeller 

Control 

\ 

Fins 

Forward  Attachment 
Point  (Frangibolt) 


Attachment  &  Release  Points 
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Forward  Section 


mmsw 


Mounting 
Interface  Ring  Bracket  (6731 

(67120)  v 


Forward 
Variable 
Ballast  Tank 
(67294) 

VB  Tank  Bracket 
(67485) 


Compass 
Pressure  Vessel 
(67106) 


LED  Plug 


Nose 

Foam 

(67698) 


Array 


Snood 

Foam 

(67117) 


Dispenser 


Acoustic 

Transponder 


*  4+  ft 


bide  Foam 
(67115) 


Snood  H 
(67107) 


Tank  Drain 


Array  Slide 
(65758) 
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Pressure  Vessel 


AFT  VB  Tank 
(68220) 


Energy  Housing 
(67599) 


Tank  Drain 


Electronics 

Support 

Structure 


Bulkhead 

(67555). 


Machined  Hull 
(67456) 


Forward 
Propulsion 
Shroud 
(67016) 

Hub 

Assembly 
(65753) 

Power  Distribution  Board  (68078) 

Electronics 
Tray  Assembly 
(68152) 


(67562) 


UNCLASSIFIED 


28 


UNCLASSIFIED 


Organizations  Involved  in  SIT 


*  Program  Executive  Office  Littoral  and  Mine  Warfare 
(PEO  LMW) 

*  Maritime  Surveillance  Systems  (PMS-485) 

'Space  and  Naval  Warfare  Systems  Center  -  San  Diego 
(SSC-SD) 

'Commander  Undersea  Surveillance  (CUS) 

'Commander,  Operational  Test  &  Evaluation  Force 

'Naval  Facilities  Engineering  Service  Center  (NFESC) 

'Johns  Hopkins  University  -  Applied  Physics  Lab  (JHU 
APL) 

'Applied  Research  Lab  -  University  of  Texas  (ARL-UT) 

*  Northrop  Grumman 

*  Lockheed  Martin 
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Organizations  Involved  in  SIT 

•  Harris  Corp 

•  Raytheon 
•EADS 
•AMRON 

•SYS  Technologies 
•Science  Applications  International  Corporation  (SAIC) 
•USNS  SIOUX  (T-ATF  171),  USNS  NAVAJO  (T-ATF  169) 
•USCGC  ASPEN 

•Sealift  Logistics  Command  Pacific 

•Fleet  Imaging  Center,  Pacific,  Combat  Camera  Group 

•Fleet  Area  Control  and  Surveillance  Facility 

•  National  Centers  for  Environmental  Prediction 
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Lessons  Learned  from  SIT 


'Developmental  Testing  should  rigorously  verify  each 
subsystem 


-The  DTV  was  tested  deeper  at  SIT  than  in  subsystem 
tests 


'Vigorously  defend  the  T&E  Program 

-Articulate  the  potential  impact  of  budget  and  schedule 
cuts 


'Augment  the  small  government  team  with  Subject 
Matter  Experts 

-  Seek  out  those  pre-eminent  in  their  field 
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NDIA  Requested  Contact  Data 


'Speaker:  Pete  Reinagel 
'Company:  SYS  Technologies 

•  Email:  peter.reinaqel@navv.mil  or 
preinaqel@svstechnologies.com 

•  Phone:  619-524-7974  or  858-232-9230  cell 
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Purpose 


•  Present  AFOTEC’s  role  in  operational  test  of 
unmanned  aerial  vehicles 
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Overview 


AFOTEC  Involvement 
Evolving  Process 
Lessons  Learned 
Future  Challenges 
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AFOTEC  Involvement 


•  AFOTEC  first  involved  when  RQ-1  was  still  an  ACTD  (’94) 
•Characterization  data  from  prior  exercises/deployments 
•Bosnia  deployments:  July  ’95  (no  AFOTEC),  March  ’96  (w/AFOTEC) 
•ACTD  has  no  formal  “user  requirements”  -  AFOTEC  adapted 
•AFOTEC  issued  first  RQ-1  report,  Nov  ‘95  W 


*  Evolution  necessary 

^Additional  ACTD’s  already  on  horizon 
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Evolving  Process 


•  Oct  ’97,  DARPA  asks  AFOTEC  to 
evaluate  Global  Hawk 

•  AFOTEC  prepared  to  add  value 


Early  Influence  mindset  evolved 
Rapidly  met  warfighter  needs 
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Lessons  Learned 


Agility  and  flexibility 
*  Adapt  tests  if  conducted  in  combat 


Test... then  deploy 

*  RQ-4  deployed  during  DT 

*  Test  vs  mission  mindset 
Surrogate  platforms 

*  Proteus  for  RQ-4 

*  Saves  time  and  money 
Early  influence 

*  Help  refine  requirements 

*  Maximize  DT/OT  interface 
Rapid  fielding 

*  Capability  to  warfighter 
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Future  Challenges 


•  National  airspace  restrictions 

•  Training  limit  =  Testing  limit 

•  Affects  all  UAVs 

•  Dedicated  test  assets 

•  Communicate  need  early 

•  CV-22  success 

•  Net-centric  solutions 

•  Architecture  important  as  vehicle 

•  Information  assurance 

•  Interoperability 

•  Improve  security 
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Summary  Slide 


*  Constant  communication  and 
coordination  are  important  for 
AFOTEC  to  stay  a  step  ahead 

-  Refine  requirements 

-  Coordinate  DT  and  OT 

*  There  will  be  more  ACTD’s  in 
the  future 


Rapid  fielding  of  effective  systems  to  allow  warfighter  to  kill 
faster  with  less  risk  to  our  Airmen,  Joint  partners,  and  Allies 
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Introduction 

New  Acquisition  Environment:  Evolutionary  Acquisition 

Development  and  Lifecycle  models 

Testing  in  Evolutionary  Acquisition  Environment 
An  Approach  to  Procurement  of  an  Autonomous  System 
Test  Design  with  Staged  Development 
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Introduction 


STM 


^  It  is  very  often  that  a  systems  engineer  faces  to  participate  in 
procurement  studies  or  make  procurement  decisions  which 
affect  significantly  the  success  of  a  project. 

^  Procurement  decisions  may 
be  complex 

involve  inputs  from  many  organizations 
include  technical  and  non-technical  constraints. 
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Introduction 


STM 

^  MAIN  AIM:  to  fulfil/achieve/realize  the  procurement  of  users  /the 
demanding  authority  in  a  reasonable  time  interval. 

^  Many  parameters 

<=>  budget 

<=>  procurement  time 

<=>  user  and/or  system  needs 

<=>  properties  of  the  system  to  be  procured 

<=>  priorities  of  the  sub-systems  present  in  the  system  (indispensable,  should 
be,  preferable) 

<=>  technological  maturity/perfection 
<=>  technologies  to  be  developed 

possibility  of  retreofil  of  the  of  the  system 
<=>  determination  of  the  tests  to  be  done 
<=>  timing  of  the  tests. 
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New  Acquisition  Environment 


STM 


^  Evolutionary  acquisition  (EA)  is  a  process  for  defense  system 
development  in  which  a  system  is  developed  in  stages  as  a 
part  of  a  single  acquisition  program. 

^  The  different  stages  can  be  additional  hardware  and  software 
capabilities  or  performance  gains  due  to  advances  in 
technological  maturity  and  reliability  growth. 
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New  Acquisition  Environment 


STM 

*■*  In  an  evolutionary  approach,  the  ultimate  capability  delivered  to 
the  user  is  provided  in  increasing  increments. 

Evolutionary  acquisition  strategies 

<=>  (1)  define,  develop,  produce  and  deploy  an  initial,  military  useful 
capability  (Increment  1)  based  on  proven  technology, 
demonstrated  manufacturing  capabilities  and  time-phased 
capabilities  needs; 

(2)  plan  for  subsequent  development,  production  and  deployment 
of  increments  beyond  the  initial  capability  over  time  (Increments  2 
and  beyond). 
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New  Acquisition  Environment 


STM 


^  There  are  various  development  and  Lifecycle  models  to  support 
systems  engineering  within  an  evolutionary  acquisition  strategy. 

waterfall 
=>  iterative 

=>  spiral  development 
=>  Vee 
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Procurement  Models 


STM 

Waterfall  Model 

'■*  It  is  a  sequential  software  development  model  (a  process  for  the 
creation  of  software)  in  which  development  is  seen  as  flowing  steadily 
downwards  (like  a  waterfall)  through  the  phases  of  project 


System  Requirements 


Analysis 


% 


Design 


Production  and  Testing 


Integration  /Deploy 


Time 
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Procurement  Models 


STM 


Iterative  Model 

Iterative  development  is  a  rework  scheduling  strategy  in  which  time  is 
set  aside  to  revise  and  improve  parts  of  the  system. 

-►  It  does  not  presuppose  incremental  development,  but  works  very  well 
with  it. 


IrrittaE 

Planning 
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Procurement  Models 

STM 

Spiral  Development  Model 

^  a  systems  development  method  used  in  information  technology. 

^  combines  the  features  of  the  prototyping  model  and  the  waterfall  model. 
'■*  favored  for  large,  expensive,  and  complicated  models 


Design 


Build 


Alternatives 


Feasftfity  Studies 
Con  steam  ts 
Design 


Evaluate 
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Procurement  Models 


Vee-Model 

*■»  a  systems  development  model  designed  to  simplify  the  understanding 
of  the  complexity  associated  with  developing  systems. 

'■*  used  to  define  a  uniform  procedure  for  product  of  project  development. 
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New  Acquisition  Environment 


STM 


'■*  All  models  provide  an  orderly  approach  to  implementing  and 
integrating  the  systems  engineering  processes  during  each 
acquisition  phase. 

Waterfall  model  is  not  practical  to  imply  for  all  systems 

=>  In  iterative  model,  the  end-state  requirement  should  be  known  but  this 
model  allows  earlier  delivery  of  initial  planned  system 

=>  In  spiral  and  incremental  development,  capability  is  developed  and 
fielded  in  increments  with  each  successive  increment  building  upon 
earlier  increments  to  achieve  an  overall  capability. 

The  spiral  and  Vee  models  rely  heavily  on  prototyping,  both  physical 
and  virtual,  to  get  user  feedback. 


2/26/2008 


13 


New  Acquisition  Environment 


STM 

Comparison  of  the  incremental  and  spiral  evolutionary  processes 


Warfighter 

Capability 


Known 


Increment 


Technology  Demos 


Feedback  from  warfighter*  — 


Not  known 
Increment 


Tlmo 
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New  Acquisition  Environment 


STM 

Evolutionary  Acquistion  is  a  strategy  not  a  model 

'■*  Can  be  adjusted  for  any  project 

'■*  General  description  of  desired  full  system  functional  capability 
^  Concise  statement  of  full  system  operational  concepts 
Flexible  overall  architecture  allowing  incremental  design 
One  method  is  the  use  of  Open  Systems  Architecture 
^  Plan  to  incrementally  achieve  desired  total  capability 

Early  definition,  funding,  development,  testing,  supporting  and 
operational  evaluation  of  initial  increment  of  operational  capability 

'■*  Continual  dialogue  and  feedback  among  users,  developers,  supporters 
and  testers 
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New  Acquisition  Environment 


STM 

^  Evolutionary  acquisition 

=>  increased  the  importance  of  traceability  in  program  management. 

►►If  a  defense  system  has  multiple  increments,  systems  engineering  can 
trace  the  evolution  of  the  system. 

►►It  can  provide  discipline  to  and  documentation  of  the  repeated  trade-off 
analyses  and  decisions  associated  with  the  program. 

Due  to  the  nature  of  evolutionary  acquisition,  design, 
development,  deployment,  and  sustainment  can  each  be 
occurring  simultaneously  for  different  system  increments. 
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Testing  in  Evolutionary  Acquisition  Environment 


STM 


^  Test  and  evaluation  may  be  at  the  end  of  the  development 
phase  which  was  the  path  in  earlier  and  simple  procurement 
models 

leads  to  a  deficiency  in  the  technological  maturity  level  as  well  as  an 
increase  in  acquisition  time  of  the  required  system. 

^  Instead  integrated  test  and  evaluation  should  be  preferred  for  the 
optimization  of  procurement  time  and  necessary  technological  maturity 
level 
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Testing  in  Evolutionary  Acquisition  Environment 

STM 

^  In  the  evolutionary  acquisition  context 

=>  experimentation  in  early  stages 

►►to  identify  system  flaws 

►►understand  the  limitations  of  system  design. 

<=>  experimentation  in  later  stages 

►►problems  identified  in  the  field  and/or  unresolved  from  earlier  testing 

►►evaluating  the  most  recent  modifications  to  the  system,  and  assessing 
the  maturity  of  a  new  component  or  subsystem  design. 

0This  experimentation  can  be  at  the  component  level,  at  the 
subsystem  level,  or  at  the  system  level,  with  varying  degrees  of 
operational  realism,  depending  on  the  goals. 
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Testing  in  Evolutionary  Acquisition  Environment 

STM 

^  Operational  testing  and  evaluation  supports  a  decision  to  pass 
or  to  fail  a  defense  system  before  it  goes  to  procurement. 

^  In  EA,  entire  spectrum  of  testing  activities  should  be  viewed  as  a 
continuous  process  of  gathering,  analyzing,  and  combining 
information  in  order  to  make  effective  decisions. 

^  The  primary  goal  of  test  programs  should  be  to  experiment, 
learn  about  the  strengths  and  weaknesses  of  newly  added 
capabilities  or  (sub)systems,  and  use  the  results  to  improve 
overall  system  performance. 

'■*  Furthermore,  data  from  previous  stages  of  development, 

including  field  data,  should  be  used  in  design,  development,  and 
testing  at  future  stages. 
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Testing  in  Evolutionary  Acquisition  Environment 

STM 

^  Operational  testing  (testing  for  verification)  of  systems  still  has 
an  important  role  to  play  in  the  evolutionary  environment, 
although  it  may  not  be  realistic  to  carry  out  operational  testing 
comprehensively  at  each  stage  of  the  development  process. 

^  Testing  early  in  the  development  stage  should  emphasize  the 
detection  of  design  inadequacies  and  failure  modes. 

^  In  evolutionary  acquisition,  it  will  be  practical  to  conduct  full- 
scale  operational  tests  only  at  stages  with  major  upgrades  or 
substantive  new  capabilities. 

^  In  the  evolutionary  acquisition  environment,  effective  system 
development  and  optimization  will  require  a  high  degree  of 
coordination  and  communication  among  system  developers, 
government  testers,  and  system  users. 
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Testing  in  Evolutionary  Acquisition  Environment 
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'■*  The  traditional  single-stage  acquisition  environment  can 
encourage  the  adoption  of  risky,  immature  technology  into  an 
existing  system,  since  it  may  take  a  decade  or  more  before  a 
new  technology  can  be  incorporated. 

^  Regardless  of  the  introduction  of  evolutionary  acquisition,  the 
increasing  complexity  of  defense  systems  implies  that  a  single 
all-encompassing,  large-scale  operational  test,  as  currently 
practiced,  will  not  be  feasible  in  many  cases. 
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Testing  in  Evolutionary  Acquisition  Environment 


STM 


^  Evolutionary  acquisition  is  being  folded  into  an  acquisition 
environment  that  already  has  a  counterproductive  incentive 
system.  The  flexibilities  inherent  in  the  evolutionary  acquisition 
process  present  even  greater  opportunities  for  these 
counterproductive  incentives  to  be  expressed. 

Testing  early  in  the  development  stage  should  emphasize  the 
detection  of  design  inadequacies  and  failure  modes.  This  will 
require  testing  in  more  extreme  conditions  than  those  typically 
required  by  either  developmental  or  operational  testing,  such  as 
highly  accelerated  stress  environments. 
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An  Approach  to  Procurement  of  Autonomous  System 


STM 


'-►Autonomous  Behavior  Technologies 
'-►Supporting  Technologies 


Perception 


Planning 


Behaviors 
and  Skills 

Autonomous 
Behavior 


Human-Robot 
Interaction 


UGV 


Learning/ 

Adaptation 
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$  Cost  of  Technology  $ 


STM 


An  Approach  to  Procurement  of  an  Autonomous 

System 


^  Field  systems  with  today’s  reliable  &  affordable  technology 
^  Insert  new  technologies  as  they  become  reliable  &  affordable 


Time 
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An  Approach  to  Procurement  of  an  Autonomous 

System 
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Robot  Autonomy 

/Intelligence 
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An  Approach  to  Procurement  of  an  Autonomous 

System 


^  The  development  process  will  be  best  served  by  systematic  and 
extensive  testing  and  refinement  under  severe  operating  conditions. 

Focusing  on  a  few  specific  applications  for  the  experimental 
prototypes,  some  of  which  may  be  simulated,  is  essential  to 
maturing  the  needed  technologies  and  resolving  the  significant 
issues  of  system  integration. 
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An  Approach  to  Procurement  of  an  Autonomous 

System 


The  user  and  developer  communities  must  work  together  to  provide 
direction  for  the  technology  integration  to  implement  vehicle 
experiments. 

^  These  directions  should  feed  into  the  spiral  development  process 
from  experimental  prototypes  to  requirements-based  systems 
following  the  established  development  process. 

=>  i.e.  application  parameters  must  be  formulated  to  address  the 

integration  of  the  mission-package  technologies,  mobility  technologies, 
and  communications  technologies  that  are  necessary  for  each 
experimental  prototype. 
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A  SIMPLE  EXAMPLE  WITH  TWO  FACTORS 

Y  =  Poo+  Pio  X1  +  &20X2  +  £ 


Stage  (k-1) 

SUNNY 

1 

VS 

RAINY 

For  details  see  “Testing  of  Defense  Systems  in  an  Evolutionary  Acquisition  Environment” 
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Test  Design  with  Staged  Development 
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Stage  (k) 


NIGHT  VISION  CAPABILITY 


Y  =  PoO+  010  *1  +  020*2  +  012*1*2  +  £ 


Test  scenarios  with  more  than  two  levels  and  quadratic  terms: 

y  =  Poo+  P 10  X 1  +  PnXi  +  p20x2  +  (321x2  +  (S12x1x2  +  £ 
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Test  Design  with  Staged  Development 


EXTENSIONS  TO  MORE  REALISTIC 

SCENARIOS 


^  New  test  scenarios 
*Y  =  f(k)(x;f3) 


f(k>  (x;P)  =  g(k)(x?;/3?) 

Effects  of  new  factors 
in  current  stage 
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Test  Design  with  Staged  Development 
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Do  full  operational  testing  and  integration  testing  only  after 
substantial  stages. 

^  Do  limited  integration  testing  at  intermediate  stages  at  which 
modifications  are  small  to  moderate. 

^  Build  in  realism  in  developmental  tests  and  carry  out  full 
component  testing  in  developmental  test. 
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Conclusion 
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*-►  Test  planning  should  be  in  the  early  stages  of  procurement 

^  Important  features  in  test  planning 

=>  procedures  to  be  followed  according  to  different  test  types 
(prototype  development  tests,  sub-system  tests,  acceptance  tests) 

=>  attandees  to  the  tests 
=>  the  criteria 
=>  evaluation 
=>  test  reports 

=>  the  conditions  for  the  next  step 
Test  structure 
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*-►  Evolutionary  acquisition  for  procurement  in  defense  systems 
=>  Evolutionary  acquisition  strategies 

►►integrate  advanced,  mature  technologies  into  producible  systems  that 
can  be  deployed  to  the  user  as  quickly  as  possible. 

►►match  available  technology  and  resources  to  approved,  time-phased, 
capability  needs. 
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Conclusion 
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^  Testing  is  an  important  milestone  for  the  procurement  of 
unmanned  vehicles 

A  test  center  for  unmanned  vehicles 
=>  Repeatable  test  scenarios 
=>  Avoid  loss  of  experience  and  knowledge 
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Substance  Field  Modeling: 
Measurement  and  Detection 

A  N# 

Air^t 

Inventioneeringcompany 

Simplify,  Perfect  and  Innovate 


Michael  S.  Slocum,  Ph.D.,  T.Sc.,  M.B.B. 


Substance-Field  Modeling 

(SFM) 


Why  A  Model  ? 


Some  systems  need  modeled  in  order  to 
understand  the  inequalities  and 
inconsistencies  that  manifest  as  problems 
or  insufficiencies 

Complex  systems  may  be  better 
understood  with  the  use  of  a  model 

A  SFM  identifies  specific  problem  types 
and  then  offers  solution  standards 


C  2008  The  Inventioneering 
Company 


SFM  Defined 


Su-Field  models  are  pictorial 
representations  of  the  elements  in  the 
system  and  the  field  through  which  the 
elements  interact. 

These  are  formed  into  triangular  diagrams 

The  diagrams  are  analyzed  for  defects 

The  particular  defect  type  yields  solution 
standards 
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Minimum  Model  of  a  Functioning 
Technical  System  (SFM) 


Object  1 ,  Substance  1 

Object  2,  Substance  2 

Field  or  force  (energy) 

Object  1  interacts  with 
Object  2  through  the 
field  or  force,  Field 

Model  system  using 
triangle 


Field 


A  complete  SFM  with  no 
defects 
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The  Model 


Substance  1  is  known  as  the  workpiece 
Substance  2  is  known  as  the  tool 
The  tool  is  applied  to  the  workpiece 

-  this  is  the  flow  of  energy  through  the  model 
The  field  is  the  connecting  component 
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The  Field 


The  Field  is  divided  into  sub-groups: 

-  Me:  mechanical 

-  Th:  thermal 

-  Ch:  chemical 

-  E:  electrical 

-  M:  magnetic 

-  EM:  electromagnetic 

The  subgroup  is  used  as  a  subscript  to  the  F 
the  Su-Field  Model  ( i.e.,  FMe,  FCh ) 
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Example 


A  person  is  painting  a  wall: 

SI:  wall  (workpiece) 

S2:  person  (tool) 

F:  painting  (mechanical  field) 


Field 


Substance  2 
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Different  Levels  of  Modeling 


SI:  wall 

•  SI:  paint 

S2:  person 

•  S2:  brush 

F:  painting 

•  F:  application 

SI:  wall 

•  SI:  brush 

S2:  paint 

•  S2:  person 

F:  adhesion 

•  F:  moving  application 

SI:  wall 

S2:  brush 

F:  application 
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76  Standard  Solutions 


The  76  Standard  Solutions  were 
developed  from  patterns  of  solutions  of 
technological  /  transactional  problems 


C  2008  The  Inventioneering 
Company 


76  Standards  of  Solutions 


The  Solutions  are  broken  into  5  Classes 

-  Class  1  -  Composition  and  Decomposition  of 
SFMs 

-  Class  2  -  Evolution  of  SFMs 

-  Class  3  -  Transition  to  Super-system  and 
Micro-level 

-  Class  4  -  Measurement  and  Detection 
Standards 

-  Class  5  -  Helpers 
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The  5  Classes 


CLASS  1.  COMPOSITION  AND  DECOMPOSITION  OF  SFMS 

GROUP  1-1 :  SYNTHESIS  OF  A  SFM 
GROUP  1-2:  DECOMPOSITION  OF  SFMS 

CLASS  2.  EVOLUTION  OF  SFMS 

GROUP  2-1:  TRANSITION  TO  COMPLEX  SFMS 
GROUP  2-2:  EVOLUTION  OF  SFM 
GROUP  2-3:  EVOLUTION  BY  COORINATING  RHYTHMS 
GROUP  2-4:  FERROMAGNETIC  SFMS  (FESFMS) 

CLASS  3.  TRANSITIONS  TO  SUPERSYSTEM  AND  MICROLEVEL 

GROUP  3-1:  TRANSITIONS  TO  BISYSTEM  AND  POLYSYSTEM 
GROUP  3-2:  TRANSITION  TO  MICROLEVEL 

CLASS  4.  MEASUREMENT  AND  DETECTION  STANDARDS 

GROUP  4-1:  INSTEAD  OF  MEASUREMENT  AND  DETECTION  -  SYSTEM  CHANGE 
GROUP  4-2:  SYNTHESIS  OF  A  MEASUREMENT  SYSTEM 
GROUP  4-3:  ENHANCEMENT  OF  MEASUREMENT  SYSTEMS 
GROUP  4-4:  TRANSITION  TO  FERROMAGNETIC  OR  OTHER  SURROGATE 
MEASUREMENT  SYSTEMS 

GROUP  4-5:  EVOLUTION  OF  MEASUREMENT  SYSTEMS 

CLASS  5.  SPECIAL  RULES  OF  APPLICATION 

GROUP  5-1:  SUBSTANCE  INTRODUCTION 
GROUP  5-2:  INTRODUCTION  OF  FIELDS 
GROUP  5-3:  USE  OF  PHASE  TRANSITIONS 
GROUP  5-4:  PHYSICAL  EFFECTS  USE 
GROUP  5-5:  SUBSTANCE  PARTICLES  OBTAINING 
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Example:  Standard  4-3-1 

If  we  are  given  the  problem  of  measurement  and 
the  problem  cannot  be  changed  to  remove  the 
need  for  measurement,  and  it  is  impossible  to 
use  copies  or  pictures,  it  is  proposed  to 
transform  this  problem  into  the  a  problem  of 
successive  detection  of  changes. 

Example:  To  measure  temperature,  it  is  possible 
to  use  a  material  that  changes  its  color 
depending  on  the  current  value  of  the 
temperature.  Alternatively,  several  materials  can 
be  used  to  indicate  different  temperatures. 

-  Uses  color  as  a  surrogate  measurement. 
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SFM  Algorithm 
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Apply  standards  from 
group  3  in  the  following 
order:  3.2.1, 3.1.1, 
_ 3.1.2.  3.1.3.  3.1.5 _ 


No 
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ISS  Solar  Panel  Array 


SFM  was 
used  to  create 
solutions  for 
surface 
imperfection 
problems  in 
the  ISS  Solar 
Panel  Array 
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SeaWolf  Submarine 


SFM  was  used 
extensively 
throughout  the 
development 
of  the  fore  and 
aft  passive 
sonar  arrays 
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Cassini-Huygens  Satellite 


SFM  was  used  in 
many  cases  to  help 
understand  complex 
system  interactions 
and  create  isolation 
systems  for  the 
Huygens  Probe 
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Conclusions 


SFM  is  a  powerful  method  of 
understanding  a  system  and  all 
interactions 

SFMs  that  contain  defects  may  be  solved 
using  the  76  Standards 

The  SFM  Algorithm  simplifies  standard 
selection 

SFM  Theory  may  be  learned  as  part  of  an 
Advanced  TRIZ  curriculum 
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You  can  learn  more  at: 

www.realinnovation.com 


www.triz-iournal.com 

'YfRIZJournal 

www.inventioneerinqco.com  and 
www.airacad.com 


A 

Air^ 

Academy 

Associates 

Simplify,  Perfect  and  Innovate 


Company 
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Class  4  -  Measurement  and 
Detection  Standards 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-1:  CHANGE  INSTEAD  OF  MEASUREMENT 

AND  DETECTION 

STANDARD  4-1-1 

If  a  problem  involves  detection  or  measurement, 
it  is  proposed  to  change  the  problem  in  such  a 
way,  so  that  there  should  be  no  need  to  perform 
detection  of  measurement  at  all. 

Example:  To  prevent  a  permanent  electric  motor 
from  overheating,  its  temperature  is  measured 
by  a  temperature  sensor.  If  to  make  the  poles  of 
the  motor  of  an  alloy  with  a  Curie  point  equal  to 
the  critical  value  of  the  temperature,  the  motor 

will  stop  itself.  C  2008  The  Inventioneering 
1  Company 


CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-1:  CHANGE  INSTEAD  OF  MEASUREMENT 

AND  DETECTION 

STANDARD  4-1-2 

If  a  problem  involves  detection  of  measurement, 
and  it  is  impossible  to  change  the  problem  to 
eliminate  the  need  for  detection  or 
measurement,  it  is  proposed  to  change/detect 
properties  of  a  copy  of  the  object  (e.g.  picture). 

Example:  It  might  be  dangerous  to  measure  the 
length  of  a  snake.  It  is  safe  to  measure  its 
length  on  a  photographic  image  of  the  snake, 
and  then  recalculate  the  obtained  result. 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-1:  CHANGE  INSTEAD  OF  MEASUREMENT 

AND  DETECTION 


STANDARD  4-1-3 

If  a  problem  involves  detection  or  measurement,  and  the  problem 
cannot  be  changed  to  eliminate  the  need  for  measurement,  and  it  is 
impossible  to  use  copies  or  pictures,  it  is  proposed  to  transform  this 
problem  into  a  problem  of  successive  detection  of  changes. 

Notes:  Any  measurement  is  conducted  with  a  certain  degree  of 
accuracy.  Therefore,  even  if  the  problem  deals  with  continuous 
measurement,  one  can  always  single  out  a  simple  act  of 
measurement  that  involves  two  successive  detections.  This  makes 
the  problem  much  simpler. 

Example:  To  measure  a  temperature,  it  is  possible  to  use  a 
material  that  changes  its  color  depending  on  the  current  value  of  the 
temperature.  Alternatively,  several  materials  can  be  used  to  indicate 
different  temperatures.  c  2008  The  lnventioneering 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-2:  Synthesis  of  Measurement  System 

STANDARD  4-2-1 

If  a  non-SFM  is  not  easy  to  detect  or  measure,  the  problem  is  solved 
by  synthesizing  a  simple  or  dual  SFM  with  a  field  at  the  output. 
Instead  of  direct  measurement  or  detection  of  a  parameter,  another 
parameter  identified  with  the  field  is  measured  or  detected.  The  field 
to  be  introduced  should  have  a  parameter  that  we  can  easily  detect 
or  measure,  and  which  can  indicate  the  state  of  the  parameter  we 


Example:  To  detect  a  moment  when  liquid  starts  to  boil,  an  electrical 
current  is  passed  through  the  liquid.  During  boiling,  air  bubbles  are 
formed  -  they  dramatically  reduce  electrical  resistance  of  the  liquid. 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-2:  Synthesis  of  Measurement  System 

STANDARD  4-2-2 

If  a  system  (or  its  part)  does  not  provide  detection  or 
measurement,  the  problem  is  solved  by  transition  to  an 
internal  or  external  complex  measuring  SFM,  introducing 
easily  detectable  additives. 


Example:  To  detect  leakage  in  a  refrigerator,  a  cooling  agent  is  mixed  with 
a  luminescent  powder.  C  2008  The  Inventioneering 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-2:  Synthesis  of  Measurement  System 

STANDARD  4-2-3 

If  a  system  is  difficult  to  detect  or  to  measure  at  a  given 
moment  of  time,  and  it  is  not  allowed  or  not  possible  to 
introduce  additives  into  the  object,  then  additives  that 
create  an  easily  detectable  and  measurable  field  should 
be  introduced  in  the  external  environment.  Changing  the 
state  of  the  environment  will  indicate  the  state  of  the 
object. 

Example:  To  detect  wearing  of  a  rotating  metal  disc 
contacting  with  another  disc,  it  is  proposed  to  introduce 
luminescent  powder  into  the  oil  lubricant,  which  already 
exists  in  the  system.  Metal  particles  collecting  in  the  oil 
will  reduce  luminosity  of  the  oil. 

C  2008  The  Inventioneering 
Company 


CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-2:  Synthesis  of  Measurement  System 

STANDARD  4-2-4 

If  it  is  impossible  to  introduce  easily  detectable  additives 
in  the  external  environment,  these  can  be  obtained  in  the 
environment  itself,  for  instance,  by  decomposing  the 
environment  or  by  changing  the  aggregate  state  of  the 
environment. 

Notes:  In  particular,  gas  or  vapor  bubbles  produced  by 
electrolysis,  cavitation  or  by  any  other  method  may  often 
be  used  as  additives  obtained  by  decomposing  the 
external  environment. 

Example:  The  speed  of  a  water  flow  in  a  pipe  might  be 
measured  by  amount  of  air  bubbles  resulting  from 

Cavitation .  c  2008  The  Inventioneering 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-3:  Improvement  of  Measurement  Systems 

STANDARD  4-3-1 

Efficiency  of  measuring  SFM  can  be 
improved  by  the  use  of  physical  effects. 

Example:  Temperature  of  liquid  media 
can  be  measured  by  measuring  a  change 
of  a  coefficient  of  retraction,  which 
depends  on  the  value  of  the  temperature. 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-3:  Improvement  of  Measurement  Systems 


STANDARD  4-3-2 

If  it  is  impossible  to  detect  or  measure  directly  the 
changes  in  the  system,  and  no  field  can  be  passed 
through  the  system,  the  problem  can  be  solved  by 
exciting  resonance  oscillations  (of  the  entire  system  or  of 
its  part),  whose  frequency  change  is  an  indication  of  the 
changes  taking  place. 

Example:  To  measure  the  mass  of  a  substance  in  a 
container,  the  container  is  subjected  to  mechanically 
forced  resonance  oscillations.  The  frequency  of  the 
oscillations  depends  on  the  mass  of  the  system. 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-3:  Improvement  of  Measurement  Systems 


STANDARD  4-3-3 

If  resonance  oscillations  may  not  be  excited  in  a 
system,  its  state  can  be  determined  by  a  change 
in  the  natural  frequency  of  the  object  (external 
environment)  connected  with  the  system. 

Example:  The  mass  of  boiling  liquid  can  be 
measured  by  measuring  the  natural  frequency  of 
gas  resulting  form  evaporation. 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-4:  Transition  to  Ferromagnetic  Measurement 

Systems 

STANDARD  4-4-1 

Efficiency  of  a  measuring  SFM  can  be 
improved  by  using  a  ferromagnetic 
substance  and  a  magnetic  field. 

Notes:  The  standard  indicates  the  use  of 
a  non-fragmented  ferromagnetic  object. 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-4:  Transition  to  Ferromagnetic  Measurement 

Systems 

STANDARD  4-4-2 

Efficiency  of  detection  or  measurement 
can  be  improved  by  transition  to 
ferromagnetic  SFMs,  replacing  one  of  the 
substances  with  ferromagnetic  particles 
(or  adding  ferromagnetic  particles)  and  by 
detecting  or  measuring  the  magnetic  field. 


C  2008  The  Inventioneering 
Company 


CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-4:  Transition  to  Ferromagnetic  Measurement 

Systems 

STANDARD  4-4-3 

If  it  is  required  to  improve  the  efficiency  of 
detection  or  measurement  by  transition  to  a 
ferromagnetic  SFM,  and  replacement  of  the 
substance  with  ferromagnetic  particles  is  not 
allowed,  the  transition  to  the  F-SFM  is  performed 
by  synthesizing  a  complex  ferromagnetic  SFM, 
introducing  (or  attaching)  ferromagnetic 
additives  in  the  substance. 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-4:  Transition  to  Ferromagnetic  Measurement 

Systems 

STANDARD  4-4-4 

If  it  is  required  to  improve  efficiency  of 
detection  or  measurement  by  transition  to 
F-SFM,  and  introduction  of  ferromagnetic 
particles  is  not  allowed,  ferromagnetic 
particles  are  introduced  in  the  external 
environment. 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-4:  Transition  to  Ferromagnetic  Measurement 

Systems 

STANDARD  4-4-5 

Efficiency  of  a  F-SFM  measuring  system 
can  be  improved  by  using  physical  effects, 
for  instance,  Curie  point,  Hopkins  and 
Barkhausen  effects,  magnetoelastic  effect, 
etc. 
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CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-5:  Evolution  of  Measurement  Systems 

STANDARD  4-5-1 

Efficiency  of  a  measuring  system  at  any  stage  of  its 
evolution  can  be  improved  by  forming  bi-  and  poly¬ 
system. 

Notes:  To  form  bi-  and  poly-systems,  tow  or  more 
components  are  combined.  The  components  to  be 
combined  may  be  substances,  fields,  substance-field 
pairs  and  SFMs. 

Example:  It  is  difficult  to  accurately  measure  the 
temperature  of  a  small  beetle.  However,  if  there  are 
many  beetles  put  together,  the  temperature  can  be 
measured  easily. 


C  2008  The  Inventioneering 
Company 


CLASS  4:  MEASUREMENT  AND 
DETECTION  STANDARDS 

GROUP  4-5:  Evolution  of  Measurement  Systems 

STANDARD  4-5-2 

Measuring  systems  evolve  towards  measuring  the 
derivatives  of  the  function  under  control.  The  transition 
is  performed  along  the  following  line: 

Measurement  of  a  function  -»  measurement  of  the 
first  derivative  of  the  function  measurement  of  the 

second  derivative  of  the  function. 

Example:  Change  of  stress  in  the  rock  is  defined  by  the 
speed  of  changing  the  electrical  resistance  of  the  rock. 


C  2008  The  Inventioneering 
Company 


UNMANNED  SYSTEMS 

T&E  Challenges  & 
Opportunities 


Dr.  James  Streilein 
February  2008 


\\ 


Agenda 


•  ATEC  Mission 

•  Achieving  Balance  (Traditional  &  Rapid) 

•  Common  Rapid  Acquisition  Challenges 

•  Unmanned  Systems  -  ATEC  Involvement 

•  Unmanned  Systems  -  Unique  Challenges 

•  T&E  Opportunities 
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ATEC  Mission: 


Does  it  Work? 
...How  Do  I  Know? 
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Mission  Statement 


•  Facilitate  equipment  procurement/fielding 
decisions  through  testing  and  analysis  to 
ensure  our  Army’s  Warfighters  have  the  right 
capabilities  for  success  across  the  entire 
spectrum  of  operations. 

•  Conduct  rapid  testing  in  direct  support  of  the 
Global  War  on  Terror,  providing  capabilities 
and  limitations  analyses  of  weapon  systems 
to  enable  employment  decisions  for  rapid 
fielding  to  the  Combat  Soldier. 
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How  We  Fit 


Under  Secretary  of 
Defense  (AT&L),  Director 
Defense  Systems/  DD 
Developmental  T&E 


DUSA 

T&E 

Executive 


Secretary  of 
the 
Army 


Chief  of  Staff 

■ 


■ 

Assistant 
Secretary  for 
AL&T 


Vice  Chief 

TEMA 

of  Staff 

/  ^||SI 

Army  Test  and 
Evaluation 
Command 

MG  Nadeau 


+ 


Army  Evaluation 
Center 

Mr.  Simmons 


Developmental 
Test  Command 

BG  Turner 


Operational  Test 
Command 

COL(P)  Potts 


Legend 

r  \ 

Independent  Reporting 

T&E  Policy 

Mandated  by  US  Code, 

and  Oversight 

OMB,  and  OSD 

^  J 

Achieving  Balance 


PAST 

•  Strong  methodical  testing 

•  Requires  Time/Money 


CURRENT 

•  Two  pronged 

•  Continue  traditional  testing 

•  Added  rapid  testing 
(Capabilities  &  Limitations) 


TRADITIONAL  ACQUISITION 

Slow,  sequential  (years) 

Risks  Mitigated 
Fully  Synchronized 
Bureaucratic 
All  data 

-  Robust  DT 

-  Full  OT 
Expensive 


RAPID  ACQUISITION 

Quick,  overlapping  (months) 
Higher  Risk 
Opportunistic 
Less  bureaucratic 
Limited  Data 

-  Constrained  DT 

-  Limited  OT 

-  C&L  Report  to  User 
Less  Expensive 
Relatively  Fast 


Common  Rapid  Acquisition 

Challenges 

•  Requirements  are  technology-driven  vice 
user-driven 

-  Often  unclear  concept  of  operations 

•  Reliance  on  contractor  test  data  and  informal 
unit  feedback 

-  Unknown  COTS  suitability  for  operational 
environmental  conditions 

•  Immature  system  supportability 
considerations  prior  to  deployment 

-  Indefinite  contractor  logistics  support 
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Unmanned  Systems 

Wide  ATEC  Involvement 


AIR 

ER/MP  (Extended  Range/Multi-Purpose) 
FCS  Class  I  (MAV  Heavy  Fuel) 

FCS  Class  IV  (Fire  Scout) 

Joint  Tactical  UAS  (Hunter) 

Small  UAS  (Raven) 

Tactical  UAS  (Shadow) 

Warrior  A 


LKMD 
MDAR 
Outpost 
SWORDS 
Spider  NMS 

ANDROS  Robot 
Concorde 

Husky  &  RG-31  Interrogation  Arms 
MV-4  Mine  Roller 
LVUSS  Robot 
MARCBot 


GMAV  (Gas  Micro  Air  Vehicle' 
Telluride 


Mini-EOD  Robot 
ODIS 

Pipe  Robots 
Rabbit 


Routerunner 
Talon  3B 

Tactibot  /  Throwbot  /  Toughbot 
xBot  (PackBot  FasTac) 


^5 

5. 

TO 


q: 


Acquisition 


Unmanned  Systems 

Unique  Challenges 


•  Increased  operator  workload 

-  Manning  &  organization  considerations 

-  Balancing  manned  and  unmanned  system  tasks 

-  Remote  spatial  orientation 

•  Complex  supportability  considerations 

-  Remote  condition  monitoring 

-  Self-recovery  and  transportability 

-  Mission  creep  increases  logistics  burden 
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Unmanned  Systems 

Unique  Challenges 


•  Weaponized  robots 

-  “Fail-safe”  design  required  for  system  safety 

-  Reliability  dependent  on  software  &  command  link 

-  Rapid  target  detection/identification  needed 

•  System -of -system  interoperability 

-  Airspace  management 

-  Spectrum  management 

-  Network  management 


10 


T&E  Opportunities 


•  Aid  requirements  and  system  development 

-  T&E  community  has  enduring,  diverse  knowledge 

•  Integrated  contractor/government  testing 

-  Learn  early;  test  within  operational  context; 
minimize  duplication 

•  Forward  Operational  Assessment  Team 

-  Feedback  loop  for  system  changes  or  updates  to 
employment  considerations  &  TTPs 

•  Mission-based  T&E 

-  System  contribution  to  mission  accomplishment 
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TEST  &  EVALUATION 


A  Critical  Enabler  to  Capabilities-Based 

Acquisition 


Dr.  James  Streilein 
Technical  Director,  ATEC 


james.streilein@us.army.mil 


703-681-9365 


GENERAL  DYNAMICS 

Land  Systems 


Application  of  MIL-HDBK-189 
Reliability  Growth  Analysis  (RGA)  on 
Mobile  Gun  System  (MGS)  Product 
Verification  Test  (PVT) 


Sharad  Kumar,  Senior  Director,  System  Engineering 
John  Paulson,  Director,  Stryker  Program 
Dmitry  E.  Tananko,  Ph.D.,  Manager,  Reliability  and  Robust 

Engineering 
Jenny  Chang,  PM  SBCT,  TACOM 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-04,  dated  02/11/08. 


Agenda 


•  What  is  MGS 

•  Success  Factors  of  MGS  PVT 

>-  Program  Management  -  Integrated  Team 
>-  System  Engineering  and  Reliability  Attainment 
►  Reliability  Data  Analysis  -  RGA 

■  FDSC  -  Failure  Definition  Scoring  Criteria 

■  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

■  Mission  Profile  and  Life  Variable 

■  Data  Grouping  and  Modeling 

■  Instantaneous  vs.  Cumulative  Reliability 

•  MGS  Lesson  Learned  -  DFR 
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GENERAL  DYNAMIC 

Land  Systems 
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Agenda 


What  is  MGS 


Success  Factors  of  MGS  PVT 


►  Program  Management  • 

►  System  Engineering  and  Reliability  Attainment 
>-  Reliability  Data  Analysis  -  RGA 

■  FDSC  -  Failure  Definition  Scoring  Criteria 

■  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

■  Mission  Profile  and  Life  Variable 

■  Data  Grouping  and  Modeling 

■  Instantaneous  vs.  Cumulative  Reliability 

•  MGS  Lesson  Learned  -  DFR 
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Land  Systems 
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Stryker  F  amily  of  V  ehicles 


_JL- 

^5"  ’"■■P 

Infantry  Carrier  Vehicle 
(ICV)  130 


Commander’s  Vehicle  Fire  Support  Vehicle 


(CV)  28 


(FSV)  14 


3$Hl 

Mobile  Gun  System  (MGS)  27 


Reconnaissance  Vehicle 
(RV)  52 


NBC  Reconnaissance  Vehicle 
(NBCRV)  3 


Medical  Evacuation  Vehicle 
(MEV)  16 


120mm  Mounted  Mortar  Carrier 
(MC-B)  37 

Engineer  Squad  Vehicle  Anti  Tank  Guided  Missile 
(ESV)  13  (ATGM)  10 


GENERAL  DYNAMIC 

Land  Systems 
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Mobile  Gun  System  -  The  Bunker  Buster 
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GENERAL  DYNAMIC 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-04,  dated  02/11/08. 


BLUF  -  Key  Factors  for  Successful 
Reliability  Growth  Program 

•  Program  Management  -  Integrated  Team 

►  The  systems,  tools,  and  practices  now  in  place  between  the  US  Government  and  General 
Dynamics  Land  Systems  allowed  the  system’s  reliability  to  grow  (repeatable  process) 

►  Reliability  growth  requires  commitments  from  Material  Developer  Team,  Combat 
Developer,  and  Independent  Test  and  Evaluation  Communities  (requirements,  test,  data, 
methodology,  tools) 

•  System  Engineering  -  Reliability  Backbone 

►  Integrates  All  Reliability  Tasks 

►  Redirects  Tasks  Toward  a  Single  Objective 

►  Crosses  Boundaries  Affecting  Operational  Reliability 

►  Provides  Program  Manager  Authority,  Funding,  and  Focus  on  Engineering,  Processes, 
Documentation,  Training,  Manufacturing,  and  Testing  for  Reliability 

•  Reliability  Data  Analysis  -  Reliability  Assessment 

►  FDSC  -  Failure  Definition  Scoring  Criteria 

►  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

►  Mission  Profile  and  Life  Variable 

►  Data  Grouping  and  Modeling 

►  Instantaneous  vs.  Cumulative  Reliability 
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GENERAL  DYNAMIC 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-04,  dated  02/11/08. 


Agenda 


•  What  is  MGS 

•  Success  Factors  of  MGS  PVT 


Program  Management  -  Integrated  Team 

►  System  Engineering  and  Reliability  Attainment 

►  Reliability  Data  Analysis  -  RGA 

■  FDSC  -  Failure  Definition  Scoring  Criteria 

■  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

■  Mission  Profile  and  Life  Variable 

■  Data  Grouping  and  Modeling 

■  Instantaneous  vs.  Cumulative  Reliability 

MGS  Lesson  Learned  -  DFR 
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GENERAL  DYNAMIC 

Land  Systems 
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MGS  Program  Management 


Plan 


•  Phase  I  -  Conduct  an  Additional 
Reliability  Test  (ART) 

►  Validate  effectiveness  of  216  PQT  and 
Post-PQT  corrective  actions 


•  Phase  II  -  Implement  changes  to 
Government  and  GDLS  Systems 
Engineering  Processes 
►  Management  and  process  changes 


•  Phase  III  -  Redesign  of  Sub-System 
components  and  integration 


Tests 

•  Additional  Reliability  Testing  (DEC  2004 
-  MAR  2005) 

►  2  vehicles 

>►  Pre-ART  -  XXX  rounds  &  X00  miles 
^  ART  -  XXX  rounds  &  X,000  miles 
>►  Reliability  Point  Estimate  XX  MRBSA 


•  Reliability  Growth  Test  (JUL-AUG  2005) 

►  2  Vehicles 
>►  XXX  rounds 
>►  X,000  miles 

>►  Reliability  Point  Estimate  XX  MRBSA 

•  Production  Verification  Testing  (APR 
2006  -  DEC  2007) 

►  3  Vehicles 

-  XXXX  rounds 
>►  XX, 000  miles 

>►  On-going  -  Current  estimate  XXX  MRBSA 


8 


GENERAL  DYNAMIC 

Land  Systems 
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MGS  Idealized  Growth  Curve 


MGS  Rebaselined  MEP  Idealized  Growth  Curve 
RGT  Demonstrated  Reliability 


Rounds  Fired 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-04,  dated  02/11/08. 


Agenda 


•  What  is  MGS 


•  Success  Factors  of  MGS  PVT 

►  Program  Management  -  Integrated  Team 

■  FDSC  -  Failure  Definition  Scoring  Criteria 

■  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

■  Mission  Profile  and  Life  Variable 

■  Data  Grouping  and  Modeling 

■  Instantaneous  vs.  Cumulative  Reliability 


•  MGS  Lesson  Learned  -  DFR 
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Land  Systems 
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MGS  -  Systems  Engineering  Approach 


•  Integrates  All  Reliability  Tasks 

•  Redirects  Tasks  Toward  a  Single  Objective 

•  Crosses  Boundaries  Affecting  Operational  Reliability 

•  Provides  Program  Manager  Authority,  Funding,  and 
Focus  on  Engineering,  Processes,  Documentation, 
Training,  Manufacturing,  and  Testing  for  Reliability 

•  Approach  Provides  Metrics  that  can  be  Measured 
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GENERAL  DYNAMIC 

Land  Systems 
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SE  Approach  to  Reliability 


Increase  Design  Effectiveness  Using 
Robust  Design  Methodology 


r  Potential  MTBF 


Modeling 
Allocation 
Prediction 
FMEA 
Parts  Program 
FRACAS 
Failure 
Prevention  & 
Review  Board 
Verification 


Design  Phase 


<4 


Manage  Growth  Potential 


Higher  Initial  MTBF 
At  Start  Of  Test 

Failure  Prevention 
Failure  Categorization 
Timely  Corrective  Actions 


RG/DT 
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Design  for  Reliability  Management 
Focuses  on  Failure  Prevention 


Requirements 

Review 


Performance 

Requirements 


Environmental* 

Requirements 


Reliability 

Requirements 

Definitions 


Safety 

Requirements 


Maintainability 

Requirements 


Support 

Requirements 


Analyses 


Testing 


Verification 


Validation 


Reliability  Growth 


IRGT,  FRACAS 


Design  for 
Reliability 

Identify 
Risk  Modes 


Update  Status 


Reliability 

a 

Growth 

A 

In  Design 

A 

rsjjJiJ/a  3iTJ d 

fisvjsyy  3osin\ 


Management  Systems 


/ 

7 

BsKrt 

[  Proca^® 

/ 

T 


Outputs,  Results,  Issues 


ipiflicijye 
liability  and 


pie 


pPfesjgn  AcxjyM 

EJfld 


GENERAL  DYNAMIC! 

Land  Systems 
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Stryker  -  Mobile  Gun  System 

Failure  Prevention  and  Resolution  Implementation 


GENERAL  DYNAMIC 

Land  Systems 
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Agenda 


•  What  is  MGS 

•  Success  Factors  of  MGS  PVT 


Program  Management  -  Integrated  Team 

System  Engineering  and  Reliability  Attainment 

Reliability  Data  Analysis  -  RGA 

■  FDSC  -  Failure  Definition  Scoring  Criteria 

■  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

■  Mission  Profile  and  Life  Variable 

■  Data  Grouping  and  Modeling 

■  Instantaneous  vs.  Cumulative  Reliability 

MGS  Lesson  Learned  -  DFR 
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GENERAL  DYNAMIC 

Land  Systems 
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Reliability  Data  Analysis 


•  Proper  Reliability  Assessment  is  a  key  for  the 
program  success  at  PVT 

•  Reliability  Assessment  must  be  discussed  up 
front  and  consensus  should  be  reached  on: 

►  FDSC  -  Failure  Definition  Scoring  Criteria 

►  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

►  Mission  Profile  and  Life  Variable 

►  Data  Grouping  and  Modeling 

►  Instantaneous  vs.  Cumulative  Reliability 
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GENERAL  DYNAMIC 

Land  Systems 
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FDSC  -  Failure  Definition  Scoring 
Criteria 

•  FDSC  is  Contractual  Document  that  defines 

►  Failure/non-Failure  Event 

►  Test  related  Event 

►  Severity  of  Failure  as  it  relates  to  the  Mission 

►  Cause  of  the  Failure 

•  FDSC  is  prepared  as  required  by  Army 
Regulation  70-1,  Army  Acquisition  Policy. 

•  FDSC  is  being  used  through  out  the  test  for 
Scoring  purposes,  hence  it  is  a  major  document 
for  Reliability  Assessment 


GENERAL  DYNAMIC 

Land  Systems 
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Failure  Categories 


•  Performance  FM  -  FM  is  repeatable  with  100%  probability  of 
failure  for  the  given  procedure/conditions.  (Example:  TDS 
overheating) 

•  Software  FM  -  same  as  above,  but  software  related. 

•  Quality  FM  -  happens  when  vehicle  is  not 
built/maintained/operated  as  designed  and  is  not  repeatable  after 
fixing  (probability  of  failure  =0%).  Can  be  broken  down  into  Initial 
Quality,  Maintenance,  Operator  error,  etc.  (Example:  Improperly 
installed  harness,  turret  lock  bended,  etc.) 

•  Potential  Reliability  FM  -  happens  when  vehicle  was 
built/maintained/operated  as  designed/intended;  probability  of 
failure  is  greater  than  0%  and  less  than  100%;  usually  happens 
due  to  wear  out,  environment,  insufficient  design,  manufacturing 
variability,  etc. 
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Failure  Mode  Categorization  Process 
Inherent  vs.  Induced  Failure 


Categorize  Failures  and  take 
Relevant  Management  Actions 


20 


GENERAL  DYNAMIC 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-04,  dated  02/11/08. 


Data  Grouping 


Known  Equivalent  Time 


Rounds 

> 



«  Actual  Mission  Profile  Run 

sit  ▲  AEC  Grouping 

Miles 

Instantaneous  MRBSA  vs  Rounds 


Crow  (NHPP) 


Rounds 


Unknown  Equivalent  Time 
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Rounds  and  Miles  Accumulation  per 
Vehicle  vs.  Calendar  Time 


Cum  IVfles  vs  Calendar  Time 


KET  Model  can  be  useful  in  the 
beginning  of  the  test  when  vehicles 
have  not  accumulated  enough 
mileage  and  rounds. 


UET  model  takes  into  account 
any  discrepancies  between 
different  vehicles  following 
through  the  test  in  calendar  time 


Calendar  Time 
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Cum.  Number  of  Failures 


Crow/AMS  AA  Model 


ReliaSoft's  RGA  6  PRO  -  RGA.ReliaSoft.com 

Cumulative  Number  of  Failures  vs  Rounds 


Rounds 


Cum  Number  of  Failures 

E(N)  =  X-TP 

Cum  Failure  Rate 

rc  = 

■EW=x.t» 

T 

Cum  MTBF 

MTBFc={rcy  ={X-T^y 

Inst  Failure  Rate 

,._d(E{N))_d{^)_,  a  tP.x 

l 

X 
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Cumulative  vs.  Instantaneous 

Reliability 

•  Reliability  growth  on  the  Development  test  is 
the  result  of  Corrective  Actions. 

•  Estimating  Reliability  of  the  product  by  taking 
the  Cumulative  reliability  (total  number  of 
failures  /  total  time  on  the  test)  does  not  take 
into  account  the  growth  on  the  test. 
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Idealized  Growth  Curve  and  Observed 
Parametric  Curve  for  Demonstrated  Instantaneous 
MRBSA 
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Agenda 


•  What  is  MGS 

•  Success  Factors  of  MGS  PVT 


Program  Management  -  Integrated  Team 
System  Engineering  and  Reliability  Attainment 
Reliability  Data  Analysis  -  RGA 

■  FDSC  -  Failure  Definition  Scoring  Criteria 

■  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

■  Mission  Profile  and  Life  Variable 

■  Data  Grouping  and  Modeling 

■  Instantaneous  vs.  Cumulative  Reliability 

IGS  Lesson  Learned  -  DFR 
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DFR  Process  Elements 


•  Boundary  Diagram  /  System  Block 
Diagram 

•  Interface  matrix 

•  P-Diagram 

•  DFMEA 

•  Reliability  &  Robustness  Metrics 

• DVP&R 

•  Reliability  Demonstration  Metrics 
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DFSS  (DCOV)  Flow  of  Analysis  &  Tools 
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Design  For  Reliability  Map 


GENERAL  DYNAMICS 

Land  Systems 
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MIL-HDBK-1 89  RGA  Method 
MGS  MEP  PVT  Instantaneous  MRBSA 


Demonstrated  Instantaneous  MRBSA  for 
decision-makers 


Crow  (NHPP) 


Failure  Rate  continues  to  decrease,  thus 
demonstrating  substantial  reliability  growth 
in  PVT 


*  Growth  Rate  is  0.4 

•  RGA  Methodology  was  developed  and 
agreed  by  RAM-T  Community 


\ 


Continuing  the  effort  to  ensure  MGS  reliability  growth 

Systems  Engineering  Process  continues  to  be  worked  “24/7” 

GDLS  Senior  Leadership  briefed  on  a  daily  basis 

Focus  on  implementation  of  Corrective  actions  on  both  the  Test 
Vehicles  and  the  Fielded  vehicles 


GDLS  teams  at  our  vendors  to  work  failure  analysis  and  ensure 
MGS  gets  their  top  priority 


Outside  experts  on  reliability  and  quality  regularly  review  our 
processes  in  engineering  and  Manufacturing  so  we  keep  getting 
better 


•  Sustained  decrease  of  MGS  Failure  Rate 
suggests  infant  mortality  region  is  passed 
and  design  is  maturing 


j 


ReliaSoft's  RGA  6  PRO  -  RGA.ReliaSoft.com 


Instantaneous  Failure  Intensity  vs. Rounds 
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Keys  to  Success 


•  Program  Management  forms  Integrated 
Team  (Material  Developers, 
Tester/Evaluators,  User)  that  has  clear 
priority  and  focus  on  Reliability  with  clear 
understanding  of  Evaluation  Criteria  and 
Test  Methods  up  front. 

•  System  Engineering  assembles  Reliability 
tools  into  Disciplined  processes  and  Working 
Organizations 

•  Reliability  Assessment  is  reached  through 
in-depth  analysis  and  consensus  between  all 
involved  parties 


r  ^ 

Program  Management  +  System  Engineering  +  Reliability  =  Success 
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Questions  and  Discussion 
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•  Dmitry  Tananko,  Ph.D. 

►  General  Dynamics  Land  Systems 

►  Tel.:  (586)  634-5071 

►  E-mail:  tanankod@qdls.com 
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24th  Annual  NDIA  Test  and  Evaluation  Conference 
25-28  February  2008,  Palm  Springs,  CA 


Applying  Design  of  Experiments 
(DOE)  methodology  to  Sortie 
Generation  Rate  (SGR)  Evaluation 


Design  of  Experiments 

m  - — #— » 


Josh  Tribble 

MILITARY  ANALYST 
AVW  TECHNOLOGIES 


Phone:  757-361-9587 
E-mail:  tribble@avwtech.com 
860  Greenbrier  Circle,  Suite  305 
Chesapeake,  VA  23320 
http://www.avwtech.com 
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Agenda 


Introduction 

-  Acquisition  humor 

-  The  Integrated  T&E  Challenge 

Intro  to  Design  of  Experiments 

SGR  Assessment  Methodology 

-  Overview  of  SGR  Assessment  to  date 

-  SGR  Assessment  objectives,  MOEs,  factors 

-  SGR  Testbed  Assessment  Design  Factors  /  Run  Matrix 

-  SGR  Live  Testing  Validation 

Benefits  of  DOE  over  single  scenario  based  analysis 
Conclusion  /  Q&A 


NOTE:  My  remarks  are  intended  to  spur  thought  on  improving  how  we  as  testers  can  do 
business  better  to  support  the  warfighter.  While  I  hope  this  aligns  well  with  DoD  and 
Services  T&E  initiatives,  I  am  not  representing  any  government  agencies’  official  position. 
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Acquisition  101? 


How  the 
user  described  it 


How 

the  requirement 
was  understood 


How  the 
contractor 
designed  it 


How  the 
programmer 
wrote  it 


How  the 
PM/sponsor 
described  it 


How  the  project 
was  documented 


What  was 
actually  installed 


How  the 

Government  was 
billed 


■ 

How  the  helpdesk 
supported  it 


What  the  user 
really  needed 


How  do  we  avoid  this? 


Integrated  T&E  Challenge 

•  Coordinated  planning  and  development  of  individual  test 
objectives 

DT  /  CT  /  OT  /  LFT&E  remain  separate  but  leverage  data 
and  resources  whenever  possible 

Potential  for  significant  cost  savings  and  earlier  risk 
reduction 

Requires  buy-in  from  all  orgs  +  strong  T&E  Working  IPT 

Requires  strong,  up-front,  test  planning  and  data  analysis 
methodology  -  Design  of  Experiments  (DOE!} 


Wf  OT&E 

kl|  DT&E 

CT 

LFT&E 


4 


Integrated  T&E 


r 

integrated  /  ~ 

^  P 


System  Disposal 

f  (CT,  DT,  OT,  LFT&E,  Joint  Exp,  M&S,  Analysis,  etc.)  dt 

Program  Conception 
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Background  of  DOE 


DOE  originated  in  the  field  of  agricultural  studies  in  the  1930s  by  R. 


Fisher,  building  on  W.T.  Gossett’s  work  at  Guinness  Brewery — Brilliant! 
Used  throughout  industry  in  industrial  experiments,  process  improvement, 
statistical  process  control 

USAF  has  significant  experience  in  use  of  DOE  across  numerous 
programs;  Navy  is  beginning  to  implement 

DOE  methodology  is  used  to  interrogate  a  process,  improve  knowledge  of 
how  the  process  works,  and  identify  factors  and  interactions  affecting 


variability  of  performance  outcomes. 
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DOE  Process  Goal  /  Benefits 


Compared  to  other  systematic  methods  DOE  designs: 


Design  of  Experiments 


u 


Yield  better  process  understanding 
Can  be  planned  and  analyzed  faster 
Cheaper  -  using  between  20-80%  of  usual  runs/tests/resources 
Better  exploration  across  range  of  performance — depth  and  breadth 
of  testing 

Challenge  assumptions  and  demonstrate  real  performance 
Better  way  to  design  and  test  complex  systems 


AVW 
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DOE  Process  Outlin 

4  Basic  Steps 


Project  description  and  decomposition 

•  Problem  statement  and  objective  of  experiment  (test) 

•  Response  variables,  and  potential  causal  variables  -  Ishikawa  fish  bone. 

Plan  test  matrix 

•  Determine  constraints,  prioritize  factors,  and  select  statistical  design  (2Kvs.  3Kvs.  mixed, 
Taguchi  vs.  classical  arrays,  full  vs.  fractional,  non-linear  effects?,  replications?,  blocking?) 

•Write  the  test  plan  with  sample  matrices,  profiles,  and  sample  output;  run  sample  analysis. 
Produce  observations  -random  run  order  &  blocked  against  unknown  effects 

•  Block  runs  to  guard  against  uncontrollable  unknown  effects  as  needed. 

Ponder  the  results 


Analyze  and  project  data;  draw  conclusions,  redesign  test  as  necessary  and  assess  results. 
Perform  “salvo  testing”  (test-analyze-test);  screen  large  #  of  factors  then  model 
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SGR  Assessment 
Methodology 
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SGR  Assessment 

Reguirements 


•  SGR  Key  Performance  Parameter 


THRESHOLD 

OBJECTIVE 

Sustained 

SGR 

Average  of  160  operational  combat 
equivalent  aircraft  sorties  in  12  hours 
of  launching  per  day  over  30  days  (26 
Flying  and  4  Non-Flying  Days  as 
specified  in  the  Design  Reference 
Mission  (DRM)  -  total  cycle  of  4160. 

Average  of  220  operational  combat 
equivalent  aircraft  sorties  with  12  hours 
of  launching  per  day  sustained  over  30 
days  (26  Flying  and  4  Non-Flying  Days 
as  specified  in  the  DRM)  -  total  cycle  of 
5720. 

Surge  SGR 

(requires  crew 
augment) 

Average  of  270  operational  combat 
equivalent  aircraft  sorties  generated 
during  each  successive  24-hour 
period  over  4  continuous  days. 

Surge:  average  of  310  operational 
combat  equivalent  aircraft  sorties 
generated  during  each  successive  24- 
hour  period  over  4  continuous  days. 

Other  Measures  of  Performance:  cycle  times,  task  timing, 
launch  and  recovery  cycles,  resource  usage,  crew  fatigue  levels,  fuel 

states/rates,  etc. 
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SGR  Assessment  Testbed 


M&S  testbed  captures  times  and  actions  associated  with  preparing, 
launching,  and  recovering  sorties  per  the  DRM 


X 

□ 

H 
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5 
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FlictitDede  L^hftnif  &  VLA 


fmtmns 
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I'J«  i 


I 


Revdlf  PtadwclctD  Ship 
aidl'td^l  W^A 
froces^rndmE 


M&S  matured  and  validated  over  time  prior  to  runs  for  score 

Live  test  used  for  validation  once  ship  is  delivered  and  aviation  certified 


SGR  Model 
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-  Mission  Planning  Timing 

-  Aircraft  Recovery  Time  Which  Encompasses: 

•  Fueling  Time 

•  Ordnance  Handling  Times 

•  Aircraft  Movement/Spotting  Times  On  The  Flight  Deck 

•  Aircraft  Movement/Spotting  Times  In  The  Hangar  Bay 

•  Aircraft  Availability 


SGR  Assessment  Analysis 

Objectives 

Determine  average  SGR  over  DRM  to  meet  KPP  requirement 
Determine  active  factors  influencing  the  variability  &  overall  outcome 

-  Measure  %  sorties  completion  rather  than  binomial  pass/fail 

-  Each  day  in  the  DRM  treated  as  a  single  design  point  due  to 
interdependencies  of  events  within  that  day 

Provide  the  fleet  with  an  analytical  model  showing  probability  of  meeting  a 
given  airplan  based  on  its  size,  mission  composition,  environment,  and 
any  other  active  factors 


%  Airplan  _  Sorties  _  Completed 


Daily  _  sorties  _  completed  _  succesfull  y 
Airplan  _  sorties 


xl00% 


Allows  equal  comparison  of  the  4  T/O  surge/sustained  requirements  across  all 
factors 

Continuous  dependent  variable  provides  more  statistical  power  than  pass/fail 
Supports  more  robust  assessment  of  capes  and  lims 


SGR  Factor  Selection 


TECHNOLOGIES 


Experimental  control  factors : 

•  Environmental 


-  Visibility/Sky  Cover:  Clear  Skies  (Case  I)  or  Clouay/iNigni  ^uase  imj 

-  Time  of  day:  midday  or  midnight  (for  12  hour  ops,  N/A  for  24  hour  ops) 


-  Sea/Winds:  state  1  vs.  3 


•  Systems: 

-  Availability:  100%  &  actual  (for  CVN-21  systems  and  aircraft) — allows  for  analysis  of 
impact  of  equipment  failures 

•  Mission 

-  Sortie  Size:  Threshold  &  Objective  levels  from  the  DRM 

-  Sustained  and  Surge  Mission  (12  vs.  24  hr  ops  (with  augmented  crew)) 

-  Operation  day:  early  and  late  in  ship  on-station  operational  period;  expect  to  interact 
with  availability  for  system  failures  and  also  translates  to  possible  crew  fatigue 

-  Airplan  mission  mix:  early/late  DRM  days  representing  different  ordnance  mix; 

-  Mission  mix  and  operation  day 


SGR  Factor  Selection  (cont’ ) 

ContmlJable  Factors  field,  constant 

•  Underway  Replenishment 

-  Not  a  factor  of  SGR  but  presumed  to  occur  on  assigned  days  or  fuel  and  ordnance  will 
not  be  available  for  the  planned  flight  days) 

•  Aircrew  augmentation 

-  Confounded  with  mission  type  -  assumed  normal  crew  for  sustained  operations  and 
augmented  crew  for  surge  missions 

Measurable  Noise  Factors 

•  Other  environmental  factors  not  controlled  (if  in  test  /  model) 

-  Temperature  extremes 

•  Specific  metrics  in  the  subordinate  models  driven  by  the  main  inputs,  such  as: 

-  Crew  fatigue  (driven  by  the  mission  day) 

-  Resource  availability 

-  Number  of  aircraft  available 

-  Weapon  skids  available 

-  Timing  for  critical  tasks,  etc. 
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SGR  Factor  Selection  (contl 
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Design  factors: 

-  Factors  with  highest  expected 
influence  listed  first 

•  Important  when  setting  up  fractional 
factorial  matrices — usually  easier  to 
resolve  factors  and  interactions 

-  Setup  for  M&S  only;  cannot  test  all 
of  these  in  live  testing 

-  Requires  M&S  improvements 

-  Need  buy-in  for  “excursions”  above 
threshold 

•  High  levels  force  the  “system” 
towards  a  higher  failure  rate  to  see 
more  variation  in  response 


Setting 

Factor 

(Low) 

-1 

(Center 

Point) 

0 

(High) 

+1 

A 

Surge/ 

Sustained 

Operations 

Sustaine 
d  (12  Hr 
ops) 

N/A 

Surge  (24 

Hr  ops 
w/augment) 

B 

Sortie  Size 
(T/0) 

Thres¬ 

hold 

Halfway 

btwn 

Objective 

C 

operational 

day 

Early  (1/4 
or  5/30) 

Mid  (2/4 
or  15/30) 

Late  (4/4  or 
26/30) 

D 

Availability 

100% 

Halfway 

btwn 

actual  /  spec 

E 

Visibility/ 

Cloud 

Cover: 

Clear/ 
Case  1 

Partly 
Cloudy/ 
Case  II? 

Cloudy/ 

Case  III 

F 

Seakeeping 

motion 

effects 

5  kts/SSI 

12 

kts/SS2 

20  kts/SS  3 

G 

Time  of  day 

Day 

Dusk? 

Night 

H 

Mission 

Day 

Early 

Mid 

late 
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SGR  Testbed  Run 
Assessment  Design 


Full  factorial  requires  28  or  256  runs 

-  Unnecessary  since  many  effects  are 
inactive 

Resulting  test  matrix  is  a  resolution  IV 
28-4  fractional  factorial  of  16  runs  +  8 
additional  runs  for  central  composite 
design 

-  Some  interactions  are  confounded  but 
can  be  resolved 

Model  DRM  days  per  the  assigned 
settings  and  evaluate  SGR  Compl  % 

“salvo  test”: 

-Runs  1-8,  then  analyze  for  effects 

-Runs  9-16,  then  reanalyze  for  effects 

-Perform  center  points  to  check  for 
linearity 

-If  necessary,  run  CCD  (face  points)  for 
non-linear  effects 


Run 

Blk 

A 

B 

c 

D 

E  = 

ABD 

F= 

ACD 

G= 

BCD 

H= 

ABC 

1 

Factorial 

1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

2 

Factorial 

1 

-1 

-1 

-1 

+1 

+  1 

+  1 

+  1 

-1 

3 

Factorial 

1 

-1 

-1 

+1 

-1 

-1 

+  1 

+  1 

+  1 

4 

Factorial 

1 

-1 

-1 

+1 

+1 

+  1 

-1 

-1 

+  1 

5 

Factorial 

1 

-1 

+  1 

-1 

-1 

+  1 

-1 

+  1 

+  1 

6 

Factorial 

1 

-1 

+  1 

-1 

+1 

-1 

+  1 

-1 

+  1 

7 

Factorial 

1 

-1 

+  1 

+1 

-1 

+  1 

+  1 

-1 

-1 

8 

Factorial 

1 

-1 

+  1 

+1 

+1 

-1 

-1 

+  1 

-1 

9 

Factorial 

2 

+  1 

-1 

-1 

-1 

+  1 

+  1 

-1 

+  1 

10 

Factorial 

2 

+  1 

-1 

-1 

+1 

-1 

-1 

+  1 

+  1 

11 

Factorial 

2 

+  1 

-1 

+1 

-1 

+  1 

-1 

+  1 

-1 

12 

Factorial 

2 

+  1 

-1 

+1 

+1 

-1 

+  1 

-1 

-1 

13 

Factorial 

2 

+  1 

+  1 

-1 

-1 

-1 

+  1 

+  1 

-1 

14 

Factorial 

2 

+  1 

+  1 

-1 

+1 

+  1 

-1 

-1 

-1 

15 

Factorial 

2 

+  1 

+  1 

+1 

-1 

-1 

-1 

-1 

+  1 

16 

Factorial 

2 

+  1 

+  1 

+1 

+1 

+  1 

+  1 

+  1 

+  1 

17 

Center  rep  1 

3 

-1 

0 

0 

0 

0 

0 

0 

0 

18 

Center  rep  2 

3 

-1 

0 

0 

0 

0 

0 

0 

0 

19 

cd  face  point  -b 

4 

-1 

-1 

0 

0 

0 

0 

0 

0 

20 

cd  face  point  +b 

4 

-1 

+  1 

0 

0 

0 

0 

0 

0 

21 

bd  face  point  -c 

4 

-1 

0 

-1 

0 

0 

0 

0 

0 

22 

bd  face  point  +c 

4 

-1 

0 

+1 

0 

0 

0 

0 

0 

23 

be  face  point  -d 

4 

-1 

0 

0 

-1 

0 

0 

0 

0 

24 

be  face  point  +d 

4 

-1 

0 

0 

+1 

0 

0 

0 

0 

AVW 


SGR  Ljve  Testing,  Validation 

Test  Design 


Live  test  conditions  and  cost  (potentially  $100M?)  limit  amount  of  live  test 
and  the  conditions 

Focus  on  validating  specific  test  points  of  interest  and  confirm  within  the 
M&S  runs  for  score 


Factor 

-1 

0 

+1 

Rationale 

A 

Surge/  Sust.  Ops 

Sustained 

N/A 

Surge 

Both  operations  can  be  run 

B 

Sortie  Size  (T/0) 

Threshold 

(T+  0)1  2 

Objective 

A  mix  of  sortie  sizes  can  be  run 

c 

Operational  day 

Early 

Mid 

Late 

No  means  of  imposing  a  late  day  due  to  cost 

D 

CVN-21/A/C  Ao 

100% 

Halfway 

Actual 

Actual  equipment  Ao 

E 

Cloud  Cover 

Actual  conditions? 

F 

Sea-State 

Actual  conditions? 

G 

Time  of  day 

Actual  condil 

ions? 

H 

DRM  Mission  mix 

Early 

j  Mid 

Late 

Factor  is  probably  inactive  so  randomly  assign 

SGR  Live  Testing  Validation 

Test  Design  (coni’) 


•  Final  Test  Matrix  with  settings: 


Test 

Case 

A:  Ops 
Type 

B:  Sortie  Level 

Actual  (# 
Sorties) 

H:  DRM 
Mission  Day 

Notes 

1 

Sustained 

Threshold 

160 

5 

Priority 

2 

Sustained 

Objective 

220 

26 

Priority 

3 

Surge 

Threshold 

270 

26 

Priority 

4 

Surge 

Objective 

310 

5 

Priority 

5 

Sustained 

Halfway  btwn 

190 

15 

Additional  run  for  midpoint 

6 

Surge 

Halfway  btwn 

290 

15 

Additional  run  for  midpoint 

7 

Sustained 

Threshold 

160 

26 

Additional  run  for  alternate  mission  mix 

8 

Sustained 

Objective 

220 

5 

Additional  run  for  alternate  mission  mix 

•  Recommend  run  during  Joint  Task  Force  Exercise  to  ensure  combat 
ready  crew  &  systems 

•  Some  analysis  of  variance  can  be  run  directly  but  main  objective  is  to 
compare  day  for  day  with  M&S  results  (including  V&V  of  lower  level 
measures  within  the  specific  process  models) 

•  Runs  1-4  are  priority;  select  additional  runs  based  on  M&S  results 
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SGR  Testbed  Assessment 

Sample  Data  Analysis 


Response  surface  plot  across  factors 
of  interest  showing  response  & 
interactions 

Table  of  plan  vs.  predicted  actual  SGR 
Completion  Rate  for  factor  settings  of 
interest  --  shows  SGR  completion  % 
falling  off  as  too  many  are  sequenced 

demonstrates  how  analysis  can 
describe  ship  caps  &  lims,  not  just  a 
pass/fail  grade  for  a  KPP  tested  only 
to  threshold 


0.0 

a 

-0.2 


-0.4 

-0.6 


-1.0 
-1.2  I 


</) 

.92  350 


300 


250 


250 


-1.2  -1.0  -0.8  -0.6  -0.4  -0.2  0.0  0.2  0.4  0.6  0.8  1.0  1.2 


Ml  0.98 
I  I  0.96 

I  I  0.94 

I  I  0.92 

Ml  0.9 
I  I  0.88 


300 


Planned  Sorties 
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CONCLUSION 


DOE  methodology: 

-may  significantly  reduce  the  required  runs  for  Testbed  Assessment  and  live  test 
validation  while... 

-providing  a  more  robust  process  for  statistical  analysis  of  variance  to  determine  where 
the  ship  design  can  and  cannot  support  a  given  air-plan  under  the  other  conditions 

-supports  robust  &  efficient  integration  ofJ/l&S  development  testing^  W&A,  & 
evaluation 


Design  of  Experiments 

m  t: — #  — »■, 


.  U 


•  DOE  is: 

-a  smarter  way  of  doing  testing 

-can  provides  superior  knowledge  to  the  systems  engineers 
-something  ail  testers  &  systems  engineers  should  become  familiar  with i 

•  QUESTIONS? 


Systems  Evaluation  of 
Unmanned  Aircraft  Systems 


(703)  681-1677 
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Agenda 


Overview  of  Current  UAS  missions 
Case  study:  Small  UAS  Evaluation 

-  OPEVAL 
-OT 

-  Lessons  Learned 

-  Doctrine  Changes 


URGENT! 

UASs  are  needed  in  the  Battle  Field 

Missions 


Need  in  OEF/OIF  to  perform  Recon  Surveil.  Target  Acq.  Missions  at  platoon 
level:  RAVEN,  DRAGON  FLY 

-  Look  for  insurgent  activity  and  provide  target  info  to  fire  support 

-  Use  in  PSY  OPS 

Retrofit  with  BAT  munitions  and  use  to  deliver  lethal  and  accurate  force: 
HUNTER  VIPER  STRIKE 

Use  as  persistent  stare  asset,  for  wide  area  surveillance  at  high  altitudes  and 
target  acquisition,  company  level  -WARRIOR  A,  GLOBAL  HAWK,  REAPER 

Use  as  signal  relay  platforms  for  connecting  to  GIG  -ERMP,  SHADOW,  ER/MP, 
FCS  Class  IV  (FIRE  SCOUT) 

Man  Un  Manned  (MUM)  Teaming  to  use  as  a  forward  asset  to  provide  target 
information  to  AH-64  -  ERMP,  HUNTER 

EOD  and  surveillance:  use  ahead  of  convoys  for  looking  at  road  details  up 
close,  platoon  level  -  g-MAV,  FCS  CLASS  I  UAS 


ARMY  UAS  PROGRAMS 


Battalion  BLOS/NLOS 
Targeting 
Comm  Relay 
Route  Recon 
Detect  Mines/CBRN 
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Overall  Evaluation  Approach: 

Start  with  ...how  is  the  USER  employing  the  UAS? 


•  Concept  of  Operations 

•  Operation  and  Organization 

•  Operational  Mode  Summary/Mission  Profile 

•  Capability  Development  Document 
(Requirements) 

•  Critical  Operational  Issues  (and  Criteria) 

•  Development  of  MOEs/MOPs/MOSs 

•  Design  of  OT  to  include  END  TO  END 
mission  accomplishment 


Case  Study:  RAVEN  UMR  and  Small 
UAS  POR  In  Support  of  Combat  Missions 


Purpose 

•  Discuss  Means  to  Assess  Capabilities  and  Limitations  on  Raven  UAS  Shipped 
for  Rapid  Deployment  to  Operations  Enduring  Freedom/Iraqi  Freedom 
(OEF/OIF) 

•  Assess  Enhancements  in  Close  Combat  Missions  Using  Small  Unmanned 
Aircraft  Systems  (SUAS) 

Data  Sources 

•  RAVEN:  Responses  from  Commanders  and  Raven  operators  in  OEF/OIF 

•  SUAS:  Production  Qualification  Testing,  Airborne  Testing  and  Initial  Operational 
Testing 

SUAS  Test  and  Evaluation 

•  Evaluate  SUAS  Effectiveness,  Suitability  and  Survivability 

•  Analyze  Changes  in  Army  Doctrine  Relevant  to  Using  the  SUAS 
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Raven  UAS  Description 


Small  all-electric  UAS 


•  Endurance  ~  60  minutes  (90  minutes  total) 

•  Range  (radio  line  of  sight)  -  10  Km 

•  Airspeed  ~  40  knots  cruise  -  52  knots 
maximum 

•  Provides  real-time  daylight  color  video  or  IR 
and  coordinate  information  and  position  of  AV 

•  Fully  autonomous  waypoint  navigation, 
automatic  altitude  and  heading  reporting 
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Raven  Capabilities/Limitations 


Enhances  Situational  Awareness 
•Reconnaissance 

•  Convoy  escort  -  Recognize  vehicles,  or  detect 
personnel  activity  next  to  a  road 

•  Point  and  route  reconnaissance,  cover  sectors  of  urban 
areas  while  patrols  go  house  to  house 

•Surveillance 

•  Cordon  and  search  operations  -  loiter  over  building(s) 
of  interest 

•  Perimeter  defense  -  detect  and  get  closer  look  at 
vehicles 

•  Target  Acquisition 

•  Use  Raven  coordinates  to  narrow  target  location  and  ID 
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Raven  Capabilities/Limitations 

•  Operators  Report  Raven  is  Simple  to  Assemble,  Program,  Launch 
and  Recover 

•  AV  parts  press  in  place  without  tools  -  ready  in  5  minutes 

•  Navigation  Waypoints  Programmed  using  MGRS 

•  No  Need  for  a  Runway  -  Small  Operations/Logistics  Footprint 

•  Damage  to  the  Air  Vehicle  Occurs  Mainly  During  Launch  and  Recovery 

•  Challenges  in  Airspace  Coordination  and  Frequency  De-confliction 

•  Extensive  Planning  and  Mission  Requests  on  Par  with  Manned  Aircraft  Missions 

•  Susceptibility  to  Visual  and  Audible  Detection 

•  Ravens  are  Difficult  to  See  and  Avoid  Due  to  Small  Size  -  Mix  with  Manned 
Aircraft 

•  Less  Noise  than  larger  UASs 
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SUAS  System  Description 


System  Description 

•  Modified  Raven  -  Improved  Visible 
Daylight  Side  View  Camera  and  IR  + 
Laser  Pointer 

•  Ground  Control  Station  and  Remote 
Video  Terminal  are  Interchangeable 

•  Raven  Air  Vehicle  Remains 
Unchanged 

•  Center  Field  of  View  Coordinates  to 
Target 


Hand-held  Video/Controller 


Air  Vehicle  (three  each) 


Ground  Control  Station 


11 


SUAS  Test  and  Evaluation 


*  SUAS  is  the  Army  Acquisition  Program  of  record  and 
Fielding  of  the  Upgraded  Raven 

*Test  and  Evaluation  Strategy 

*  Effectiveness 

•Answer  Critical  Operational  Issue  (COI)  “Does  the  SUAS  Enhance 
the  Combat  Effectiveness  of  a  Small  Unit?” 

•Leads  to  Comparative  Analysis  of  Close  Combat  Missions  with  and 
Without  the  SUAS 

•Force  on  Force  Test  Examines  Use  of  the  SUAS  During  Attacks, 
Raids,  Convoy  Escort,  and  Observation  of  Known  Areas  of  Interest 

•Evaluates  Technical  Requirements  the  System  Must  Meet  (Range  to 
Target,  Interoperability,  and  Capability  to  Detect/Recognize  Vehicles 
and  Personnel) 

•Data  Obtained  from  Initial  Operational  Test  (IOT) 
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SUAS  Test  and  Evaluation 


*Test  and  Evaluation  Strategy  (Continued) 

•  Suitability/Survivability 

•Answers  Critical  Operational  Issue  “Does  the  SUAS  Support  the  Small 
Unit’s  Sustained  Operational  Requirements?” 

•Required  Reliability  Testing,  Portability,  Transportability,  Airborne 
Capability  MANPRINT,  and  Supportability  Assessments 

•System  Reliability  Data  Obtained  From  Production  Qualification  Testing, 
and  IOT 

•Portability  Assessed  through  dismounted  operations  During  IOT 

•During  the  Logistics/MANPRINT  Demonstration  Verified/Validated 
Technical  Manuals,  Training,  Assembly/Disassembly,  Preventive 
Maintenance  Checks  and  Services  (PMCS),  MOPP  IV  and  Cold  Weather 
Tasks 

•Electromagnetic  Environmental  Effects  (E3)  Data  Collected  from 
Production  Qualification  Testing 

•System  Susceptibility  to  Acoustic  Signature  obtained  from  Developmental 
Test  and  IOT 
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Approach  to  Evaluating  Effectiveness 

•  Initial  Operational  Test  (IOT) 

•  Mission  Matrix  Consists  of  21  Infantry  Company  Level  Close  Combat  missions 

•  Nine  Pairs  of  Comparison  Missions  with  and  without  the  SUAS 

•  Three  to  Meet  Data  Requirements  According  to  the  OMS/MP  and  for  the 
Purpose  of  RAM 


MISSION 

Daytime 

Nighttime 

With 

SUAS 

Without 

SUAS 

With 

SUAS 

Without 

SUAS 

Conduct  a  Convoy  Escort  (ARTEP  71-2-2320) 

1 

1 

1 

1 

Conduct  an  Attack  by  Fire  (ARTEP  71-2-0219) 

1  and  (1) 

1 

1 

1 

Conduct  a  Raid  (ARTEP  71-2-0308) 

1  and  (1) 

1 

2  and  (1) 

2 

Defend  a  Battle  Position  (ARTEP  71-2  2603) 

1 

1 

1 

1 
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Approach  to  Evaluating  Effectiveness 


•  Comparative  Test  With  and  Without  the  SUAS  Required  Using  a 
Robust  OPFOR  and  Cluttered  Environment  to  Challenge  the  Test 
Unit 

•  Mission  Effectiveness  Templates  (MET)  and  Assessment  of  Mission 
Outcome  by  Subject  Matter  Experts 

•  METs  use  Timeliness,  Accuracy,  and  Unit  Resources  as  Attributes  to 
Objectively  Score  Missions  With  and  Without  the  SUAS 

•  ARTEP  MTPs  Assessed  by  O/Cs  to  Observe  Accomplishment  of  Task  Steps 

•  The  Entire  Mission  is  Timed  by  Distinct  tasks,  SALUTE  Reports  are  Recorded 
and  Actions  the  Unit  Takes  to  Complete  the  Mission 

•  SMEs  Individually  Assess  the  Mission  as  it  Unfolds  in  the  Battlefield  and  Based 
on  Mission  Outcome  Determine  Mission  Success  and  Commander’s  Intent 


ARTEP=Army  Training  And  Evaluation  Program 
SALUTE=Size  Activity  Location  Unit  Time  Equipment 
MTP=Mission  Training  Plan 
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Approach  to  Evaluating  Effectiveness 


CC  =  Casualty  count  factor;  CC  =  R  -  2(B  +  NC) 
R:  %  OPFOR  killed  or  captured 
B:  %  BLUFOR  killed  or  captured 
NC:  %  NON  COMB,  killed  or  captured 


16 


Approach  to  Evaluating  Effectiveness 
(Mission  Effectiveness  Templates) 


Timeliness  (30%)  (T)  Score 

Planning  (40%) 

Mission  Planning  Time  (20%) 

Time  to  Initial  SALUTE  Report  (80%) 

Execution  (60%) 

Mission  Execution  Time  (LD)  (100%) 

Accuracy  (50%)  (A) 

Battalion  PIRs  (30%) 

Number  of  PIRs  Detected*  (50%) 
Number  of  PIRs  Recognized  (50%) 

Planning  (40%) 

Man-sized  Object  Detections  (50%) 
Locations  (20%) 

Vehicle  Recognitions  (30%) 

Execution  (30%) 

Man-sized  Object  Detections  (50%) 
Locations  (20%) 

Vehicle  Recognitions  (30%) 


100 

0 

Actual 

30 

240 

X  0.024 

20 

240 

X  0.096 

30 

240 

X  0.180 

Detections 

0 

X  0.100 

Recognitions 

0 

X  0.100 

Detections 

0 

X  0.100 

Recognitions 

0 

X  0.040 

Recognitions 

0 

X  0.060 

Detections 

0 

X  0.100 

Recognitions 

0 

X  0.040 

Recognitions 

0 

X  0.060 

PIR=  Priority  Information  Requirements 
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Approach  to  Evaluating  Effectiveness 
(Mission  Effectiveness  Templates) 


Company  Resources  (20%)  (R)  Score  100  0 


Planning  (40%) 

Personnel  (50%)  2  15 

Vehicles  (50%)  0  2 

Execution  (60%) 

Personnel  (50%)  2  15 

Vehicles  (50%)  0  2 


Actual 

X  0.040  = 
X  0.040  = 

X  0.060  = 
X  0.060  = 


MET  Score  =  T  +  A  +  R 


•Total  Scores  Without  SUAS  Compared  to  Total  Scores  With  SUAS  Using  a  t- 
distribution  Paired  Samples  Test  and  Wilcoxon  Signed  Rank  Test 
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Approach  to  Evaluating  Effectiveness 


Results  of  Mission  Effectiveness  Template  Scores  by  Mission 

With  SUAS 

Without  SUAS 

Attack 

Day 

34.9 

34.1 

Night 

35.4 

19.8 

Raid 

Day 

24.2 

47.9 

41.9 

Night 

_ 36.3 

Convoy  Escort 

Day 

40.5 

Night 

41.5 

Defense  (Observe  NAI) 

Night 

38.4 

20.2 

Note:  Scores  highlighted  in  red  have  a  zero  rating  due  to  an  unsuccessful  mission  outcome 


Statistical  Analysis  Using  a  Paired  Samples  Test  and  Wilcoxon  Signed  Ranks  Test 

Paired  Samples  Test 

Wilcoxon  Sign-Rank  Test 

t-Ratio 

0.777403 

Degrees  of  Freedom 

7 

Test  Statistic 

5 

Prob  p  >  |t| 

0.4624 

Prob  p  >  |z| 

0.547 

Probp  >t 

0.2312 

Prob  p  >  z 

0.273 
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Loss  Exchange  Ratios 


EH 

Losses  With  SUAS 

Losses  Without  SUAS 

Day/Night 

OPFOR 

BLUFOR 

Non- 

Combatants 

Score 

OPFOR 

BLUFOR 

Non-  .  1 

Com&atants 

Score 

Convoy 

operations 

Day 

0% 

ss 

b.6% 

55.8 

88.9% 

0.0% 

0.0% 

100.0 

Convoy 

operations 

Night 

0% 

6.o% 

55.8 

50.0% 

9.1% 

0.0% 

71.6 

Hasty 

Attack 

Dav 

353% 

0.0% 

0.0% 

mrm 

■fill 

40.0% 

9.7% 

10.5% 

55.5 

Hasty 

Attack 

mm 

81,8% 

4.1% 

36.8% 

55.8 

0 

0.0% 

H 

'  55.8 

Raid  to 
capture 

Day 

80.0% 

0.0% 

15.4% 

80.3 

75.0% 

10.5% 

0.0% 

82,6 

Raid  to 
capture 

Day 

85.7% 

35.7% 

wmmmm 

1 

62,9 

70.6% 

11.1% 

67.8 

■ 

jjmm | 

30.8% 

0.0% 

0.0% 

71.1 

55.6% 

. 

2,6% 

81.3% 

6.6 

Table  6.  Loss  exchange  ratios  on  missions  with  and  without  the  SUAS 


Critical  T-value=  1.943,  thus  no  statistical  significance 


Approach  to  Evaluating  Effectiveness 


Evaluation  of  SUAS  Effectiveness 

•  SUAS  Demonstrated  Negligible 
Enhancements  in  Combat  Effectiveness  at 
the  Company  Level 

•  SUAS  Enhanced  Situational  Awareness 
Based  on  Closed  Set  of  Close  Combat 
Mission  Iterations  -  used  to  confirm  course 
of  action 

•  MET  Scores  Show  No  Significant 
Differences  in  Mission  Outcome 

•  System  Interoperability  Proven  in  95%  (25 
of  26)  Successful  Downloads  from  the  GCS 
to  the  Laptop 

•  SUAS  Demonstrated  Insufficient  Capability 
to  be  used  in  Call  for  Fire  Tasks 

•  AV  Flown  Mostly  at  Lower  Altitudes  to 
Recognize  Personnel 


MGRS:  13S  CR  57763  56937 


Alt:  4422  ft  MSL 

Mag:  148° 
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Approach  to  Evaluating  Suitability/ 

Survivability 

Reliability  Availability  and  Maintainability 

•  Did  Not  Meet  MTBOMF  for  sub-systems:  The  AV  is  a  risk  item  with  much  lower 
MTBOMF  than  anticipated 

•  Battery  issues  caused  crashes 

•  Bent  motor  shafts  from  failed  launches 

Portability 

•  Demonstrated  Rucksack  Portability  Weight  <  25  lbs  to  Include  Single  Air  Vehicle 

Capability  For  Airborne  Operations  -  Static  Line  and  HALO  Jumps  Met 
Launch  and  Recovery  without  the  Need  for  a  Runway 

•  Twenty  one  of  61  Launches  Resulted  in  Failure  -  Attributed  to  Nighttime  Slight 
Wind  Conditions 

•  Emphasize  Operator  Training  on  Launch  Techniques  - 

Navigation,  Operation  and  Recovery  Operations  -  Mounted/ 
Dismounted 

Flight  Endurance  -  Did  not  Show  it  could  Meet  the  90-minute  Endurance  22 


Approach  to  Evaluating  Suitability/ 

Survivability 


•  Commander/Operator  Workload 

•  Emphasize  Commander  Participation  During  Collective  Training  -  Delegate  Use  of 
the  System  to  the  XO,  or  FSO  to  Focus  on  Key  Missions  Tasks 

•  System  Susceptibility  to  Loss  of  Link  -  demonstrated  in  10  of  28 
Missions 

•  AV  is  audible  at  Mission  Altitude  -  Opposing  Force  (OPFOR)  changed 
mode  of  operation 
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Changes  in  Army  Doctrine 
by  Integrating  the  SUAS 

•  Doctrine  -  ways  to  a  means 

•  New  weapon  systems  can  instill  revisions  in  Army  doctrine  -  ways  to  conduct 
combat  by  reducing  the  size  of  the  force 

•  SUAS  Tradeoffs  to  Reduce  Size  of  Offensive  Force 

•  Traditional  Offensive  Operations  Count  on  a  Three  to  One  Offensive  Force 

•  Cannot  Violate  Principles  of  War  -  mass,  surprise,  economy  of  force,  maneuver 

•  SUAS  must  demonstrate  capability  as  a  force  multiplier  to  enhance  small  unit 
effectiveness 

•  Improve  sensor  technology  to  identify  individuals  and  the  location  of  Improvised 
Explosive  Devices  (IED) 

•  Enhance  interoperability  to  network  with  other  UASs  and  UGVs  -  increase  sensor 
coverage  over  target 

•  SUAS  is  a  good  test  case  for  Future  Combat  use  of  small  UAVs 

•  Defines  the  technical  envelope  to  look  forward  and  meet  small  unit  Aerial  RSTA 
needs 
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Conclusions 


•  Raven  -  Operational  history  shows  adequate  RSTA  task  capability  in 
limited  Close  Combat  Missions  at  the  Infantry  Company  Level 

•  SUAS  incorporates  minor  upgrades  to  Raven  UAS 

•  Targeting  Feature  is  a  step  in  the  right  direction  -  not  capable  now 

•  Requires  stable  platform  and  sensor 

•  Benefits  to  the  Small  Unit 

•  Light  Infantry  Unit  may  make  better  use  of  the  SUAS  than  say  a  mechanized 
Infantry  Unit  -  lack  of  high  power  sensors  limit  capabilities  in  open  terrain 

•  Emphasize  the  Need  for  Powerful  Sensors 

•  System  risk  to  detection  offers  the  opposing  forces  advantages  to  change  modes 
of  operation 
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National  Test  and  Evaluation  UV  Conference  Brief 

RDML  Stephen  S.  Voetsch 

February  2008 
UNCLASSIFIED 


COMOPTEVFOR 


Who  we  are 

•  Overview 

•  Mission  Based  Test  Design 

•  Integrated  Test 

•  Open  Architecture 

•  Unmanned  Vehicles 

•  History  -  Present  -  F 

•  OPTEVFOR  Role  in  Capab 
&  Requirements 

•  Discussion 


Pax  River 

Norfolk 

New  River 


2 


Note:  Dotted  line  =  coordination 


Workforce 


*  5  Man-years 


□  Officer 


■  Enlisted 


□  Civil  Service 


□  Contractor 


COTF  Headquarters  Total:  296 
(Does  not  include  approx  135  Program  Funded  Contractors) 

As  of  01 DEC2007 


Portfolio 


Budget 


A 

$11. 11V 

r 

<ur 

1  2M 

y 

■  Office  of  Naval 

Research 

□  Program 

Funding 


FY07:  $51 .3M 
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Undersea  Warfare 

ARC-l 

VA  Class  ESM  System 
MK48  ADCAP 
SSGN  Class 
VA  Class 
ISIS 

Photonics  Mast 

Surface  Warfare 

Aegis  Ballistic  Missile  Defense 

USCG  Multi-Mission  Coastal  Helo 

SSDS 

CEC 

Evolved  Sea  Sparrow 
CIWS  Block  IB 
DDG  1000 
LPD-17 

Rolling  Airframe  Missile 

Tomahawk 

DDG  51 

LCS 

SPY-ID 

T-AKE 

LHA-6 


Portfolio 


Air  Warfare 
EA-18G  Growler 
FA-18  ATFLIR 
AESA 

FA-18E/F  H4E 

AH-1W  Helo  Night  Targeting 
System 
AIM-9X 

Improved  Extended  Echo 
Ranging  System 
IDECM 
JMPS 
JSOW-C 
V-  22 
VH-71 

H-1  Upgrade 
JSF 

SLAM  ER 

JDAM 

USQ-113 

EA-6B 

CVN-21 

JPATS 

AARGM 


C4I 

ERP 

Tactical/Mobile 

NGEN 

Maritime  Domain  Awareness 

SEWIP 

GCCS-M 

Wideband  Global  SATCOM 
CSEL 

JICO  support  System 
Net  Centric  Enterprise 
Services 

Expeditionary  Warfare 

JBPDS 

JCAD 

JEOD 

JSAM 

SRDRS 

Sea  Eagle  ACTD 
JWARN 

Joint  Effects  Model 


Program  list  not  all  inclusive 
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Navy  Initiatives 

>  The  Navy’s  Provider  Enterprise 

>  (T&E  Board  of  Directors) 

>  T&E  Cycle  Time  Reduction 

>  Mission  Based  Test  Design 

>  Integrated  Testing 
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Single  Process  Owner 


“The  need  for  a  T&E  capability  that  is 
synchronized  with  product  procurement  across 
the  spectrum  is  critical  to  support  the  Navy’s 
future  war  fighting  needs.  As  part  of  the 
enterprise  integration  of  T&E  there  is  a  need  for 
a  single  Navy  T&E  process  owner.  An  additional 
dimension  is  the  need  for  a  more  effective  and 
efficient  business  model  in  the  relationship 
between  government  and  industry,  the  desire  to 
achieve  synergism,  and  produce  a  “win-win  ” 
situation  for  both  entities.” 

COMOPTEVFOR  “Streamline  T&E”  Report  to  CNO  July  2005 
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T&E  Board  of  Directors 


Ms  Margret  Harrell  SES  FFC 
Program  Executive  Office 
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Mission  Based  Test  Design 
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Integrated  Test 


Integrated  Test 


“Integrated  testing  is  the  collaborative 
planning  and  collaborative  execution  of  test 
phases  and  events  to  provide  data  in  support 
of  independent  analysis,  evaluation  and 
reporting  by  all  stakeholders  particularly  the 
developmental  (both  contractor  when 
appropriate  and  government)  and  operational 
test  communities.” 
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Operational  Test  and  Evaluation  Force 


National  Test  and  Evaluation  UV  Conference  Brief 

RDML  Stephen  S.  Voetsch 


February  2008 
UNCLASSIFIED 


enterprise 


Knowledge  Management  and 

Retention 

A  looming  crisis  for  the  unprepared 

Presented  by 
VSE  Corporation 


'RATION 


Enterprise 

Excellence 


Knowledge  Management  and  Retention 


Historical  Attrition 

■  Competition 

■  Retirement 

■  Career  Change 

■  Catastrophic  Accident 


'RATION 


Enterprise 

Excellence 


Knowledge  Management  and  Retention 


Issues  of  the  21st  Century 


■  Advanced  Technology 

■  Specialized  Knowledge 

■  Baby  Boomer  Generation 

■  Uneven  Hiring  Practices 


Expectation:  30%  to  50%  Attrition  within  5  Years 


'RATION 


Enterprise 

Excellence 


Knowledge  Management  and  Retention 


Knowledge  at  Risk: 


■  Historical  Products  Still  Supported 

■  Networks  and  Contacts  for  Problem-Solving 

■  Complex  Systems  Corrective  Actions 

■  Unique  Diagnostic  Capabilities 


Usually  Not  Adequately  Documented 


'RATION 


Enterprise 

Excellence 


Knowledge  Management  and  Retention 


Why  Is  It  Allowed  to  be  Lost? 

■  Critical  Knowledge  (CK)  not  Documented 

■  CK  not  Shared 

■  CK  not  Recognized  as  Critical  or  Unique 

■  Time  Required  to  Document  or  Share 

■  The  Most  Critical  is  Usually  the  Most  Complex 


Not  a  Matter  of  Risk  -  A  Certainty 


'RATION 


Enterprise 

Excellence 


Solving  the  Problem 


Step  1 :  Identify  the  CK  in  the  Workforce 
Leading  Questions: 

■  What  are  our  core  competencies? 

■  Why  have  we  been  able  to  succeed? 

■  What  sets  us  apart  from  the  competition? 

■  What  do  we  do  better  than  anyone  else? 


'RATION 


Enterprise 

Excellence 


Solving  the  Problem 


Step  2:  ID  Present  Method  of  Storing  CK 

■  May  Be  Adequate 

■  Improvement  may  suffice 

■  Major  Action  May  Be  Needed 


'RATION 


Enterprise 

Excellence 


Solving  the  Problem 


Step  3:  Identify  Who  Has  the  CK 


■  Who  Do  Managers  Rely  on  Most? 

■  Can  Alternates  Perform  the  Function  As  Well? 

■  Peer  Reviews 

■  Who  Do  Employees  Seek  Out? 
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'RATION 


Enterprise 

Excellence 


Solving  the  Problem 


Step  4:  Develop  a  CK  Storage  System 


■  Many  Options  Available 

■  Must  Be  Tailored  to  Organization’s  Culture 

■  Must  Be  Efficient 

■  Critical  Balance 
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'RATION 


Enterprise 

Excellence 


Solving  the  Problem 


Step  5:  Gather  &  Store  CK 
Difficulty  Dependant  On: 

■  Organizational  Culture 

■  Amount  of  Segmentation 

■  Time  Available 

■  Resources  Allocated 

■  Number  of  Individuals  with  CK 

■  Amount  of  CK 


'RATION 


Enterprise 

Excellence 


Expected  Roadblocks 


The  Knowledge  Miser 


■  Status  Achieved  Through  CK 

■  May  Be  Reluctant  to  Relinquish  Status 


11 


'RATION 


Enterprise 

Excellence 


Knowledge  Miser  Solution 


KM’s  Interests  Addressed  Through: 


■  Org.  Ability  to  Maintain  Long-Term  Benefits 

■  Potential  Near-Term  Layoffs 

■  Status  Through  Knowledge  Sharing 
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'RATION 


Enterprise 

Excellence 


Expected  Roadblocks 


The  Distrustful: 


■  Feels  ill-treated  by  Organization 

■  Resentful 

■  No  desire  to  help  organization 
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'RATION 


Enterprise 

Excellence 


Expected  Roadblocks 


The  Distrustful  Solutions: 


■  Identify  Source  of  Resentment 

■  Re-Establish  Positive  Relationship 

■  Assign  to  Project  Teams 
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'RATION 


Enterprise 

Excellence 


Expected  Roadblocks 


The  Contractor 


■  CK  Increases  in  Value  Near  Retirement 

■  CK  Considered  a  Commodity 

■  Give  Away  Now  or  Sell  Later 

■  Only  Delays  the  Problem 
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'RATION 


Enterprise 

Excellence 


Expected  Roadblocks 


Contractor  Solutions: 


■  Exchange  CK  Prior  to  Return  as  Contractor 

■  Initial  Assignment:  Clarify  process  & 
Methodologies 
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'RATION 


Enterprise 

Excellence 


Knowledge  Management  and  Retention 


Knowledge  Management  Systems 

■  Training 

•  Structured  process 

•  Results  measured  &  evaluated 

•  Significant  resource  requirements 
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Storytelling 

•  Informal,  casual 

•  Social  activity 


'RATION 


Enterprise 

Excellence 


Knowledge  Management  and  Retention 


Knowledge  Management  Systems 

■  Coaching 

•  Provided  as  needed 

•  Growth  process 
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Exit  Interview 

•  Most  commonly  used 

•  Least  effective 


'RATION 


Enterprise 

Excellence 


Knowledge  Management  and  Retention 


Knowledge  Management  Systems 

■  Critical  Incident  Report 

•  Easy  implementation 

•  Limited  application 

■  Network  of  Specialists 

•  Serious  cross-training 

•  Mutual  support 
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'RATION 


Enterprise 

Excellence 


Knowledge  Management  and  Retention 


Knowledge  Management  Systems 

■  Exit  Interview 

•  Most  commonly  used 

•  Least  effective 


■  Mentoring 

•  More  structured  than  coaching 

•  Dependent  on  favorable  relationships 

•  Most  effective,  most  challenging 
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'RATION 


Enterprise 

Excellence 


CK  Management  &  Retention 

Summary 


■  Match  Culture,  System,  and  Resources 

■  System  Must  Be  User  Friendly 

■  Resource  Requirements  Increase  with  Time 

■  Management  Choice: 

•  Develop  a  System  that  Captures  and  Retains  CK 

•  Re-create  CK  at  a  later  date 

•  Lose  the  capability 
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Army  Test  and  Evaluation  Command 


Agenda 


•  Accomplishments 

•  Four-Element  Framework  (4-EF)  Overview 

•  Improvements  made  to  the: 

♦  Mission  Element 

♦  System  Element 

♦  Mission  to  System  Interface 

♦  Evaluation  Element 

♦  Test  Element 

•  Detailed  Development  of  the  4-EF  Execution 

•  Mission-Based  Test  and  Evaluation  Strategy 

•  Summary 
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Army  Test  and  Evaluation  Command 


A  ccomplish  m  ents 

•  Briefed  at  23rd  NDIA  National  T&E  Conference,  13  March  2007 

•  CH-47F  Case  Study  (completed) 

♦  Applied  4-EF  to  the  evaluation  of  the  CH-47F. 

♦  Modified  the  construct  of  the  all  four  elements  (mission,  system,  evaluation 
and  test). 

♦  Verified  the  modified  element  designs  by  producing  a  4-EF  designed 
System  Evaluation  Plan  (SEP). 

•  Joint  Cargo  Aircraft  Case  Study  (ongoing) 

♦  Started  application  of  4-EF  to  the  evaluation  of  the  JCA. 

♦  Validated  design  of  the  mission  and  system  elements  by  producing  them  . 

♦  Continuing  to  develop  evaluation  and  test  elements  on  road  to  producing  4- 
EF  designed  SEP. 

•  Held  Mission-Based  T&E  Strategy  Summit  (5th-6th  Feb  08) 

♦  Path  Ahead:  Integrate  -  Demonstrate  -  Coordinate 
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4-EF  Overview 


Elements 

•  Mission 

♦  Mission  tasks  and  sub-tasks. 

•  System 

♦  System  items,  functions  and  characteristics. 

•  Evaluation 

♦  Evaluation  measures. 

•  Test 

♦  Data  sources  and  products. 


ELEMENTS  AND  INTERFACES 


TEST  ELEMENT 

Data  Sources  and  Data  Products 


MISSION  ELEMENT 

Mission  Tasks  and  Sub-tasks 

> 

> 

V  A 

i  V 

Mission  to  System  Interface 

Mission  to  Evaluation  Interface 

A  A 

Y  v 

SYSTEM  ELEMENT 

System  Functions  &  Characteristics 

$ 

System  to  Evaluation  Interface 

T 

EVALUATION  ELEMENT 

System  Performance  (MOPs)  and  Operational  Performance  (MOEs) 

Measures 

A 

V 

Evaluation  to  Test  Interface 

Interfaces 

•  Mission  to  System 

♦  How  the  system  supports  the  mission  tasks. 

♦  Gives  operational  conditions  and  linkages. 

•  Mission  to  Evaluation 

♦  How  operational  tasks  are  evaluated. 

♦  Gives  operational  conditions. 

•  System  to  Evaluation 

♦  How  system  performance  is  evaluated. 

•  Evaluation  to  Test 

♦  How  the  data  supports  the  evaluation. 

Traces 

•  Planning  starts  at  the  mission  element,  progresses  through  the  system  and  evaluation  elements  and  ends  at  the  test 
element. 

•  Execution  of  the  T&E  effort  starts  at  the  test  element,  progresses  through  the  evaluation  and  system  elements  and 
ends  at  the  mission  element. 
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Mission  Element 


Purpose 

•  To  describe  unit  mission  and  tasks 
required  to  accomplish  that  mission. 

Components 

•  Task:  A  task  is  defined  as  a  discrete  action 
that  the  unit  (system  and  its  operators)  must 
perform  in  order  to  accomplish  its  mission. 
For  example:  communicate  with  ground  unit. 

•  Task  Levels:  Orderly  breakdown  of  the 
mission  into  tasks  and  sub-tasks. 


r 


v 


Three  types  of  tasks  were  identified  in 
order  to  assess  impact  of  the  task  on 
overall  mission  performance. 

•  Mission  Execution  Tasks 

•  Conditional  Mission  Tasks 

•  Mission  Enabling  Tasks 


A 


J 


COI:  How  capable  is  the  (unit  and  system)  in  supporting 
(mission  statement)  in  an  operational  environment? 


Mission 


Taskl 


Task  2 


1st  Sub-task 
of  Task  2 


2nd  Sub-task 
of  Task  2 


Level  0  (Mission) 


1st  Sub-task 
of  Task  2.2 


2nd  Sub-task 
of  Task  2.2 

Alternate  2nd 
Sub-task  of 
Task  2.2 


Level  1  (Task) 


I . ►  Taskn 


Level  2  (Task) 

nth  Sub-task 
of  Task  2 


3rd  Sub-task 
of  Task  2.2 


Level  3  (Task) 


nth  Sub-task 
of  Task  2.2 


Army  Test  and  Evaluation  Command 


Mission  Element 

Task  Type  Definitions 


Mission  execution  tasks. 

♦  Tasks  that  describe  a  discrete  action  that  the 
unit  (system  and  its  operators)  must 
perform  in  order  to  accomplish  its  main 
mission. 

♦  Examples:  Navigate,  ID  Target,  Send  BDA 
message,  etc. 

Conditional  mission  tasks. 

♦  Tasks  that  are  performed  during  the  mission 
that  become  required  due  to  some 
influencing  condition.  They  are  not 
normally  required  to  successfully  execute 
the  mission. 

♦  Examples:  Avoid  threat  missile,  Extinguish 
engine  fire,  etc. 

Mission  enabling  tasks. 

♦  Tasks  that  enable  the  mission  execution  and 
conditional  tasks  to  be  performed.  They 
usually  occur  before  or  after  the  mission. 

♦  Examples:  Training,  Maintain,  etc. 


1.  When  aggregated  along  with  system  enabling  characteristics. 


Mission  Enabling  Tasks 

Level  1  Tasks  ,  Level  2  Tasks 


hH 

i-K 


3d  in  Fly  to  Destination  task  in  ess 


A 


j 

~\ 


J 


Aggregated 

into 

Effectiveness 


> 


Aggregated 

into 

Suitability1 
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System  Element 


Purpose 

•  To  describe  the  system  and  the  system 
functions  and  characteristics. 

Components 

•  System  Items:  Makeup  of  the  system  and 
sub-systems. 

•  System  Functions:  Description  of  the 
function  an  item  must  perform  in  support  of 
the  mission. 


Added  system  “characteristics”  to 
accommodate  suitability  attributes 
(reliability,  maintainability,  etc.). 


•  System  Characteristics:  Description 
of  a  particular  quality  of  the  system 
that  affects  whether  the  item  can 
Yjerform  a  function. 


1st  System 


Characteristic  of 
the  System 


Function  of  the 
Integrated  System 


1st  Sub-system 
of  System  1 


Function  of  the 
Sub-system 


2nd  Sub-system 
of  System  1 

Characteristic  of 
the  Sub-System 


Function  of  the 
Sub-system 


2nd  System 


Characteristic  of 
the  System 


Sub-system  of 
System  2 


Characteristic  of 
the  Sub-System 


2nd  Characteristic 
of  Sub-System 

Function  of  the 
Sub-system 


Component  of 
System  2 


System  of  Systems 


3rd  System 


Characteristic  of 
the  System 


Characteristic  of 
the  System 


Level  1 

(System/Sub-system) 


Sub-system  of 
System  3 

1st  Function  of 
Sub-system 

2nd  Function  of 
Sub-system 


Function  of  the 
Sub-system 


Level  2 

(Sub-system/Component) 


Army  Test  and  Evaluation  Command 


Mission  to  System  Interface 


Purpose 

•  To  describe  how  the  mission  tasks  relate  to  the 
system  attributes  (functions  and  characteristics). 


Components 

•  Input  Rule:  Description  of  how  the  system 
items  relate  to  the  mission  task.  Uses  logical 
input  rules,  such  as  AND  and  OR  to  describe 
links  to  more  than  one  system  or  function. 


•  Conditions:  Description  of  the  physical, 
military,  and  civil  variations  that  affect 
performance  of  a  task.  Comprise  the  operational 
test  conditions  used  in  DT  and  OT.  For  example; 
weather  conditions,  countermeasures,  urban 
environment,  etc. 


Some  system  characteristics  will  affect 
all  tasks.  These  are  labeled  “mission 
enabling  characteristics”  and  are 
aggregated  with  mission  enabling 
tasks  to  assess  suitability. 
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Evaluation  Element 


Purpose 

•  To  describe  the  evaluation  measures  and  how  they  relate  to  mission  tasks  and  system  attributes. 
Components 

•  Standard:  Acceptable  performance  of  the  system  attribute  or  mission  task  in  terms  of  the  MOE/MOP. 

•  Link  to  System-focused  COI/C:  Column  in  the  evaluation  element  that  identifies  which  MOE/Ps 
are  used  to  evaluate  the  system-focused  COI/C. 


Re-aligned  evaluation  measures  to 
distinguish  evaluation  of  system 
performance  from  task  capabilities 
and  facilitate  planning. 

•  Measure  of  Effectiveness  (MOE): 
Measure  used  to  evaluate 
operational  capability  (task 
capability). 

•  Measure  of  Performance  (MOP): 
Measure  used  to  evaluate  system 
attribute  performance. 


COI:  Does  the  (system)  perform  (system  capability)? 

Task  1 

1st  Sub¬ 
task  of 
Task  2 

1st  Sub¬ 
task  of 
Task  2.2 

Standard 

for 

MOP 

MOP  for  Characteristic 
of  System  1 

Characteristic 
of  System  1 

Input  Rule 
Conditions 

Input  Rule 
Conditions 

Input  Rule 
Conditions 

Standard 

for 

MOP 

1st  MOP  for  Function  of 
Sub-system  1.1 

Function  of 
Sub-system 

1.1 

Input  Rule 
Conditions 

Input  Rule 
Conditions 

Standard 

for 

MOP 

2nd  MOP  for  Function  of 
Sub-system  1.1 

X 

Standard 

for 

MOP 

MOP  for  Characteristic 
of  System  2 

Characteristic 
of  Sub-system 
2.2 

Input  Rule 
Conditions 

Input  Rule 
Conditions 

Standard 

for 

MOP 

MOP  for  Function  of 
Sub-system  2.2 

Function  of 
Sub-system 
2.2 

Input  Rule 
Conditions 

Standard 

for 

MOP 

MOP  for  Function  of 
Sub-system  2.2 

Function  of 
Sub-system 
2.2 

Input  Rule 
Conditions 

Standard 

for 

MOE 

MOE  for  Sub-task  of  Task  2 

Conditions 

Standard 

for 

MOE 

MOE  for  Sub-task  of  Task  2.2 

Conditions 

Army  Test  and  Evaluation  Command 


System  Evaluation  Plan 

Evaluation  Planing  Construct 


Current  Evaluation  Construct 


4-EF  Evaluation  Construct 


ESS 


COI/Cs 


MOE 


MOP 


All  results  aggregated  into: 

-  is/is  not  effective, 

-  is/is  not  suitable,  and 

-  is/is  not  survivable. 


Capabilities  and  limitations  presented  at  the 
mission  level. 

Can  be  aggregated  into  ESS. 

Effectiveness  =  execution  &  conditional  tasks 
Suitability  =  enabling  tasks  &  characteristics. 


ESS  evaluation  based  on  critical 
issues  and  criteria. 


Mission  capabilities  and  limitations  based  on 
execution  of  ALL  tasks  necessary  to 
accomplish  the  mission. 

COI/Cs  addressed  through  evaluation  of  tasks. 
-  COIs  are  a  sub-set  of  tasks  or  system 
function/characteristics. 


MOEs  are  a  roll-up  of  MOPs. 

-  Can  be  operational  in  nature. 

-  Sometimes  technical  in 

nature  in  order  to  support  criterion. 

-  Not  necessarily  “measures”  since 
they  are  aggregations. 


MOEs  are  a  measure  of  task  accomplishment. 

-  Always  operational  in  nature. 

-  Truer  to  DAU  definition  of  a  MOE;  “Measure 
designed  to  correspond  to  accomplishment  of 
mission  objectives  and  achievement  of 
desired  results.” 


MOPs  are  tested  and  evaluated. 

-  Sometimes  technical  in  nature 

-  Sometimes  operational  in 
nature.  (OT  measures). 


•  MOPs  are  a  measure  of  system  performance. 

-  Always  technical  in  nature. 

-  Truer  to  DAU  definition  of  a  MOP;  “Measure 
of  a  systems  performance  expressed  as 
speed,  payload,  range,  time  on  station, 
frequency,  or  other  distinctly  quantifiable 
performance  features.” 


•  Lower  level  tasks  evaluated  to  determine 
operational  and  system  performance  on 
overarching  task. 

-  Allows  T&E  at  right  level  of  fidelity. 


Mission 


Level  1  Task 


MOE 


System  &  Function/ 
Characteristic 


MOP 


Level  2  Task 
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Test  Element 


Purpose 

•  To  describe  the  data  products,  the 
sources  of  the  data  products,  and 
how  they  relate  to  the  evaluation 
element’s  MOPs. 

Components 

•  Link  to  MOPs:  Description  of 
which  data  products  support  which 
MOPs. 

•  Data  Products:  Specific  data  packet 
obtained  though  a  data  source 
satisfying  a  MOP  data  requirement. 

•  Data  Sources:  The  specific  source 
of  a  data  product. 


Time  Phased 


Contractor 

Test 

M&S 

Developmental 

Test 

OT 

Event  #1 

M&S 

OT 

Event 

#2 

M&S 

DATA 

SOURCE 

Data  Product  #1 

Data  Product  #2 

Data  Product 

Data  Product  #1 

Data  Product  #2 

Data  Product  #3 

Data  Product  #1 

Data  Product  #2 

Data  Product 

Data  Product 

Data  Product 

X 

MOP 

MOP 

X 

X 

MOP 

X 

X 

MOP 

X 

X 

MOE 

X 

MOE 

f  \ 

Test  element  “time  phased”  to  facilitate  assessment  of  T&E  program  and  to  describe  an 

integrated  T&E  plan  where  the  most  appropriate  data  is  used  at  the  most  appropriate 

time. 

V  J 
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System  Evaluation  Plan 

Test  Planning  Construct 
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4-EF  Execution 

(AKA  “The  3  Questions ”) 


Army  Test  and  Evaluation  Command 


4-EF  Execution  Example 


SEP 


Level  1  Task 

Fly  to  a  Destination. 


MOE:  Ability  to  operate 
navigation  equipment. 


System:  Avionics 
Function:  Provide 
navigation  and 
communication  functions. 


MOP:  Compliance 
with  global  air  traffic 
management. 


Test 


Task  performance 
measured 
against  standard  in 
SEP. 


System 
performance 
measured 
against  standard  in 
SEP. 


Evaluation 


Combined  task 
capabilities/ 
limitations. 


Task  capabilities/ 
limitations. 


Task  capabilities/ 
limitations  from  system 
performance. 


Level  2  Task 

i  i 

i  i 

Lower  level  task 

i 

i 

i  i 

1  "V 

!  ^ 

1 .  Crew  can  self-deploy  x,xxx  nm. 

Deploy  Aircraft. 

i  i 

capabilities/  limitations. 

i 

i  i 

i 
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Example 


1 .  Crew  is  able  to  accurately 
navigate  the  aircraft. 

2.  Crew  can  self-deploy  x,xxx  nm. 

3.  Crew  can  operate  in  worldwide 
airspace  except  for  regions  A 
and  B. 

4.  Crew  is  restricted  to  VFR  flight 
in  regions  A  and  B. 


1 .  Crew  is  able  to  accurately 
navigate  the  aircraft. 


1 .  Crew  can  operate  in  worldwide 
airspace  except  for  Europe  and 
North  Africa. 

2.  Crew  is  restricted  to  VFR  flight 
in  regions  A  and  B. 


1.  Complies  with  worldwide  B- 
RNAV,  P-ILS  and  Mode  S. 

2.  ATC  radios  do  not  provide 
fractional  frequency  spacing. 
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4-EF  Execution 


Mission  Capability  Scale  Ongoing  Development 

•  Each  mission  task  is  examined  in  two  areas: 

♦  Does  the  system  support  task  accomplishment, 
and 

♦  Does  the  task  support  mission  objectives  & 
required  capabilities? 


•  Rating  scale  is  similar  to  Cooper-Harper  or 
Bedford  Workload  Scales. 


•  Rating  scale  not  sequential  in  order  to  reflect 
higher  importance  of  task  meeting  mission 
objectives  vs.  system  meeting  task 
requirements. 


•  Lower  the  number,  the  better  the  mission 
performance. 


f  .  \ 

Concept  being  assessed  for: 

•  Functionality:  Can  the  ATEC  system  team  apply  the  methodology? 

I  •  Quality:  Are  the  results  repeatable  and  meaningful?  I 


Task  accomplished  fully  as  anticipated. 


All  aspects  of  mission  objectives  are  achievable 
(commander’s  intent,  mission  timeline,  resources 
required,  etc.). 


Task  accomplished  only  through  workarounds 
or  with  increased  operational  cost  (manpower, 
schedule,  equipment,  etc.). 

Minimal  operational  costs.  All  aspects  of  mission 
objectives  are  achievable  (commander’s  intent, 
mission  timeline,  resources  required,  etc.). 

2 

Substantial  operational  costs.  Some  aspects  of 
mission  objectives  are  degraded. 

5 

High  operational  costs.  Some  or  all  aspects  of 
mission  objectives  are  not  achievable. 

8 

Task  accomplished  under  some  conditions  only. 


All  aspects  of  mission  objectives  still  achievable 
(commander’s  intent,  mission  timeline,  resources 
required,  etc.). 

3 

Some  aspects  of  mission  objectives  are  degraded. 

6 

Some  or  all  aspects  of  mission  objectives  are  not 
achievable. 

9 

Task  not  accomplished  under  any  conditions. 

All  aspects  of  mission  objectives  still  achievable 
(commander’s  intent,  mission  timeline,  resources 
required,  etc.). 

4 

Some  aspects  of  mission  objectives  are  degraded. 

7 

Some  or  all  aspects  of  mission  objectives  are  not 
achievable. 
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Mission-Based  T&E  Strategy 

Study  Group 

•  Study  group  was  formed  in  early  December  2007.  Participants  included: 

♦  U.S.  Army  Test  and  Evaluation  Command, 

♦  U.S.  Army  Evaluation  Center, 

♦  U.S.  Army  Research  Laboratory  -  Survivability/Lethality  Analysis  Directorate, 

♦  U.S.  Army  Research  Laboratory  -  Human  Research  &  Engineering  Directorate, 

♦  U.S.  Army  Materiel  Systems  Analysis  Activity,  and 

♦  DOT&E,  Joint  Test  and  Evaluation  Methodology 

•  The  group  examined  four  major  efforts  in  detail:  Four  Element  Framework 
(AEC),  Net  Enabled  Battle  Command  (ATEC),  Missions  and  Means 
Framework  (ARL),  Capability  Test  Methodology  Measures  Framework 
(JTEM). 

•  Approach: 

♦  Integrate  the  various  concepts  with  the  goal  of  developing  a  consistent, 
repeatable  and  robust  integrated  T&E  methodology  for  evaluating  the 
capabilities,  limitations  and  contributions  of  networked  system  of 
systems  in  improving  a  U.S.  force’s  ability  to  accomplish  their  assigned 
tasks. 
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Mission-Based  T&E  Strategy 
Study  Group  Results 

•  Results  Achieved 

♦  Group  Consensus:  An  evaluation  focused  on  the  mission  tasks  will  show 
causality  of  individual  system  performance  on  capabilities  and  limitations 
provided  by  the  system-of-systems  (to  include  the  contribution  of  the 
networked-enabled  command  and  control). 

♦  Group  Focus:  Development  of  integrated  mission-based  T&E  process. 
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Mission-Based  T&E  Strategy 

Path  Ahead 

Mission-Based  T&E  Strategy  Summit  held  5th-6th  Feb  08. 

♦  Briefed  out  study  group  results  to  senior  steering  group. 

♦  T&E  focused  on  the  mission  tasks  is  correct  path  ahead 

♦  Future  efforts  should  focus  on... 

»  Development  of  a  common  definition  of  terms:  Common  language  needs  to 
be  developed  to  ensure  understanding  across  organizations. 

•  Demonstration  of  the  integrated  process:  The  process  needs  to  be 
demonstrated  to  validate  the  concept  and  to  show  that  the  anticipated  benefits 
can  be  realized. 

B  Coordination  of  the  concept  with  the  rest  of  the  acquisition  community: 

Concept  is  best  executed  with  a  coordinated  effort  between  requirements 
generator,  materiel  developer,  and  T&E  Community.  Also,  lessons  learned 
should  be  shared  with  others  working  similar  concepts. 


18 


Army  Test  and  Evaluation  Command 


Summary 

•  Four-Element  Framework  has  evolved  through  case  study 
application  of  the  methodology. 

♦  More  robust  mission  element. 

♦  Suitability  attributes  addressed. 

♦  System  evaluation  plan  format  developed. 

♦  Test  element  re-designed  to  facilitate  assessment  of  T&E 
strategies. 

♦  Methodology  for  mission  task  capability  roll-up  developed. 

•  Path  ahead  for  mission-based  T&E  lies  in  coordination 
with  the  acquisition  community. 

♦  Development  of  common  terms. 

♦  Demonstration  of  an  integrated  process. 

♦  Open  discussions  with  requirement,  materiel  developer,  and  T&E 
community. 
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Acronym  Chart 


AA 

Additional  Attribute 

MOP 

Measure  of  Performance 

AKA 

Also  Known  As 

OA 

Operational  Area 

ATC 

Air  Traffic  Control 

OT 

Operational  Test 

AV 

All  View  (slide  4) 

OT&E 

Operational  Test  and  Evaluation 

AV 

Air  Vehicle  (slides  11,  13,  andl5) 

OV 

Operational  View 

CDD 

Capabilities  Development  Document 

RSTA 

Reconnaissance,  Surveillance  &  Target  Acquisition 

COI 

Critical  Operational  Issue 

RT 

Remote  Terminal 

CPD 

Capabilities  Production  Document 

SATCOM 

Satellite  Communications 

DAG 

Data  Authentication  Group 

SV 

Systems  View 

DoD 

Department  of  Defense 

T&E 

Test  and  Evaluation 

DT 

Developmental  Test 

T/O 

Takeoff 

GCS 

Ground  Control  Station 

TM 

Telemetry 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

TV 

Technical  View 

KPP 

Key  Performance  Parameter 

UAS 

Unmanned  Aerial  System 

MER 

Mission  Evaluation  Report 

UAV 

Unmanned  Aerial  Vehicle 

METT-TC 

MOE 

Mission,  Enemy,  Terrain,  Troops,  Time  and  Civil 

Measure  of  Effectiveness 

VFR 

Visual  Flight  Rules 
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